vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0245.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

165 lines
18 KiB
HTML
Raw Blame History

<a name="dli_08_0245"></a><a name="dli_08_0245"></a>
<h1 class="topictitle1">RDS Sink Stream</h1>
<div id="body1574393091541"><div class="section" id="dli_08_0245__en-us_topic_0111499975_section848675393917"><h4 class="sectiontitle">Function</h4><p id="dli_08_0245__en-us_topic_0111499975_p18442315202820">DLI outputs the Flink job output data to RDS. Currently, PostgreSQL and MySQL databases are supported. The PostgreSQL database can store data of more complex types and delivers space information services, multi-version concurrent control (MVCC), and high concurrency. It applies to location applications, financial insurance, and e-commerce. The MySQL database reduces IT deployment and maintenance costs in various scenarios, such as web applications, e-commerce, enterprise applications, and mobile applications.</p>
<p id="dli_08_0245__p19151352874">RDS is a cloud-based web service.</p>
<p id="dli_08_0245__en-us_topic_0111499975_p14652141216256"></p>
<p id="dli_08_0245__en-us_topic_0111499975_p32841975">For more information about RDS, see the <em id="dli_08_0245__en-us_topic_0111499975_i842352697185442">Relational Database Service User Guide</em>.</p>
</div>
<div class="section" id="dli_08_0245__en-us_topic_0111499975_section1773605853811"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0245__en-us_topic_0111499975_ul810551215398"><li id="dli_08_0245__en-us_topic_0111499975_li6425436163814">Ensure that you have created a PostgreSQL or MySQL RDS instance in RDS.<p id="dli_08_0245__en-us_topic_0111499975_p104771013203812"><a name="dli_08_0245__en-us_topic_0111499975_li6425436163814"></a><a name="en-us_topic_0111499975_li6425436163814"></a>For details about how to create an RDS instance, see <span class="filepath" id="dli_08_0245__en-us_topic_0111499975_filepath99491231102211"><b>Creating an Instance</b></span> in the <em id="dli_08_0245__en-us_topic_0111499975_i41691210192314">Relational Database Service User Guide</em>.</p>
</li><li id="dli_08_0245__en-us_topic_0111499975_li14111191212396">In this scenario, jobs must run on the dedicated queue of DLI. Therefore, DLI must interconnect with the enhanced datasource connection that has been connected with RDS instance. You can also set the security group rules as required.<p id="dli_08_0245__p94091618103916"><a name="dli_08_0245__en-us_topic_0111499975_li14111191212396"></a><a name="en-us_topic_0111499975_li14111191212396"></a>For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_08_0245__dli_08_0249_en-us_topic_0132966325_filepath13726182511815"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_08_0245__dli_08_0249_cite534516e564224b13bad1de4471da3139181527">Data Lake Insight User Guide</cite></i>.</p>
<p id="dli_08_0245__p17953337436">For details about how to configure security group rules, see <span class="filepath" id="dli_08_0245__dli_08_0249_filepath78235157113"><b>Security Group</b></span> in the <em id="dli_08_0245__dli_08_0249_i082519153111">Virtual Private Cloud User Guide</em>.</p>
</li></ul>
</div>
<div class="section" id="dli_08_0245__en-us_topic_0111499975_section5292631103113"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0245__screen10868159131716"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">stream_id</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;rds&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">db_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span>
<span class="w"> </span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<p id="dli_08_0245__en-us_topic_0111499975_p997205394019"></p>
</div>
<div class="section" id="dli_08_0245__section2068851615195"><h4 class="sectiontitle">Keywords</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0245__en-us_topic_0111499975_table1648420306385" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0245__en-us_topic_0111499975_row248543016387"><th align="left" class="cellrowborder" valign="top" width="11.75%" id="mcps1.3.4.2.2.4.1.1"><p id="dli_08_0245__en-us_topic_0111499975_p3485130143810">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.54%" id="mcps1.3.4.2.2.4.1.2"><p id="dli_08_0245__en-us_topic_0111499975_p168561350102814">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="78.71000000000001%" id="mcps1.3.4.2.2.4.1.3"><p id="dli_08_0245__en-us_topic_0111499975_p148513023812">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0245__en-us_topic_0111499975_row19485193073810"><td class="cellrowborder" valign="top" width="11.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0245__en-us_topic_0111499975_p3485330133817">type</p>
</td>
<td class="cellrowborder" valign="top" width="9.54%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0245__en-us_topic_0111499975_p08567508283">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="78.71000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0245__en-us_topic_0111499975_p13485123063812">Output channel type. <strong id="dli_08_0245__en-us_topic_0111499975_b842352706193629">rds</strong> indicates that data is exported to RDS.</p>
</td>
</tr>
<tr id="dli_08_0245__en-us_topic_0111499975_row9485143010387"><td class="cellrowborder" valign="top" width="11.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0245__en-us_topic_0111499975_p1748515305387">username</p>
</td>
<td class="cellrowborder" valign="top" width="9.54%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0245__en-us_topic_0111499975_p1385665012811">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="78.71000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0245__en-us_topic_0111499975_p548513301385">Username for connecting to a database.</p>
</td>
</tr>
<tr id="dli_08_0245__en-us_topic_0111499975_row6485163083810"><td class="cellrowborder" valign="top" width="11.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0245__en-us_topic_0111499975_p1248511308389">password</p>
</td>
<td class="cellrowborder" valign="top" width="9.54%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0245__en-us_topic_0111499975_p78566504284">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="78.71000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0245__en-us_topic_0111499975_p148573019388">Password for connecting to a database.</p>
</td>
</tr>
<tr id="dli_08_0245__en-us_topic_0111499975_row648533013820"><td class="cellrowborder" valign="top" width="11.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0245__en-us_topic_0111499975_p248510306381">db_url</p>
</td>
<td class="cellrowborder" valign="top" width="9.54%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0245__en-us_topic_0111499975_p10856145082816">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="78.71000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0245__en-us_topic_0111499975_p14859300387">Database connection address, for example, <strong id="dli_08_0245__en-us_topic_0111499975_b842352706192752">{database_type}://ip:port/database</strong>.</p>
<p id="dli_08_0245__en-us_topic_0111499975_p193531452515">Currently, two types of database connections are supported: MySQL and PostgreSQL.</p>
<ul id="dli_08_0245__en-us_topic_0111499975_ul18228175913311"><li id="dli_08_0245__en-us_topic_0111499975_li122812591335">MySQL: 'mysql://ip:port/database'</li><li id="dli_08_0245__en-us_topic_0111499975_li112281159133316">PostgreSQL: 'postgresql://ip:port/database'</li></ul>
</td>
</tr>
<tr id="dli_08_0245__en-us_topic_0111499975_row2661452184316"><td class="cellrowborder" valign="top" width="11.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0245__en-us_topic_0111499975_p18662052154317">table_name</p>
</td>
<td class="cellrowborder" valign="top" width="9.54%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0245__en-us_topic_0111499975_p1585655032811">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="78.71000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0245__en-us_topic_0111499975_p1566312520436">Name of the table where data will be inserted.</p>
</td>
</tr>
<tr id="dli_08_0245__en-us_topic_0111499975_row123510295489"><td class="cellrowborder" valign="top" width="11.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0245__en-us_topic_0111499975_p11506104112011">db_columns</p>
</td>
<td class="cellrowborder" valign="top" width="9.54%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0245__en-us_topic_0111499975_p95068492017">No</p>
</td>
<td class="cellrowborder" valign="top" width="78.71000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0245__en-us_topic_0111499975_p95061743206">Mapping between attributes in the output stream and those in the database table. This parameter must be configured based on the sequence of attributes in the output stream.</p>
<p id="dli_08_0245__en-us_topic_0111499975_p165848247233">Example:</p>
<pre class="screen" id="dli_08_0245__en-us_topic_0111499975_screen28841856122619">create sink stream a3(student_name string, student_age int) with (
type = "rds",
username = "root",
password = "xxxxxxxx",
db_url = "mysql://192.168.0.102:8635/test1",
db_columns = "name,age",
table_name = "t1"
);</pre>
<p id="dli_08_0245__en-us_topic_0111499975_p19798110172017">In the example, <strong id="dli_08_0245__en-us_topic_0111499975_b8423527062051">student_name</strong> corresponds to the name attribute in the database, and <strong id="dli_08_0245__en-us_topic_0111499975_b84235270620512">student_age</strong> corresponds to the age attribute in the database.</p>
<div class="note" id="dli_08_0245__en-us_topic_0111499975_note187991037132716"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0245__ul5885228355"><li id="dli_08_0245__li17885728653">If <strong id="dli_08_0245__b10239144763514">db_columns</strong> is not configured, it is normal that the number of attributes in the output stream is less than that of attributes in the database table and the extra attributes in the database table are all nullable or have default values.</li></ul>
</div></div>
</td>
</tr>
<tr id="dli_08_0245__en-us_topic_0111499975_row6584144623319"><td class="cellrowborder" valign="top" width="11.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0245__en-us_topic_0111499975_p5917239431">primary_key</p>
</td>
<td class="cellrowborder" valign="top" width="9.54%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0245__en-us_topic_0111499975_p14912236436">No</p>
</td>
<td class="cellrowborder" valign="top" width="78.71000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0245__en-us_topic_0111499975_p1251143121719">To update data in the table in real time by using the primary key, add the <strong id="dli_08_0245__en-us_topic_0111499975_b84235270616106">primary_key</strong> configuration item (<strong id="dli_08_0245__en-us_topic_0111499975_b2042108653161034">c_timeminute</strong> in the following example) when creating a table. During the data writing operation, data is updated if the specified <strong id="dli_08_0245__en-us_topic_0111499975_b84235270616118">primary_key</strong> exists. Otherwise, data is inserted.</p>
<p id="dli_08_0245__en-us_topic_0111499975_p12866182174316">Example:</p>
<pre class="screen" id="dli_08_0245__en-us_topic_0111499975_screen15277181717477">CREATE SINK STREAM test(c_timeminute LONG, c_cnt LONG)
WITH (
type = "rds",
username = "root",
password = "xxxxxxxx",
db_url = "mysql://192.168.0.12:8635/test",
table_name = "test",
primary_key = "c_timeminute");</pre>
</td>
</tr>
<tr id="dli_08_0245__en-us_topic_0111499975_row119481740175619"><td class="cellrowborder" valign="top" width="11.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0245__en-us_topic_0111499975_p1794917404566">operation_field</p>
</td>
<td class="cellrowborder" valign="top" width="9.54%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0245__en-us_topic_0111499975_p15949840175610">No</p>
</td>
<td class="cellrowborder" valign="top" width="78.71000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0245__en-us_topic_0111499975_p10949194018561">Processing method of specified data in the format of ${field_name}. The value of <strong id="dli_08_0245__en-us_topic_0111499975_b61442012314">field_name</strong> must be a string. If <strong id="dli_08_0245__en-us_topic_0111499975_b912971311616">field_name</strong> indicates D or DELETE, this record is deleted from the database and data is inserted by default.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0245__section956515719204"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0245__en-us_topic_0111499975_p13386336154418">The stream format defined by <strong id="dli_08_0245__en-us_topic_0111499975_b842352706193745">stream_id</strong> must be the same as the table format.</p>
</div>
<div class="section" id="dli_08_0245__en-us_topic_0111499975_section171023793119"><h4 class="sectiontitle">Example</h4><p id="dli_08_0245__en-us_topic_0111499975_p12360185814444">Data of stream <strong id="dli_08_0245__en-us_topic_0111499975_b84235270614114">audi_cheaper_than_30w</strong> is exported to the <strong id="dli_08_0245__en-us_topic_0111499975_b84235270694052">audi_cheaper_than_30w</strong> table in the <strong id="dli_08_0245__en-us_topic_0111499975_b842352706141114">test</strong> database.</p>
<div class="codecoloring" codetype="Sql" id="dli_08_0245__en-us_topic_0111499975_screen15458151114815"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">audi_cheaper_than_30w</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_price</span><span class="w"> </span><span class="nb">INT</span>
<span class="p">)</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;rds&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;root&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;xxxxxx&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">db_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;mysql://192.168.1.1:8635/test&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;audi_cheaper_than_30w&quot;</span>
<span class="p">);</span><span class="w"> </span>
</pre></div></td></tr></table></div>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0240.html">Creating a Sink Stream</a></div>
</div>
</div>
View Git Blame Copy Permalink