forked from docs/doc-exports
Su, Xiaomeng
76a5b1ee83
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
181 lines
21 KiB
HTML
181 lines
21 KiB
HTML
<a name="dli_08_0247"></a><a name="dli_08_0247"></a>
|
|
|
|
<h1 class="topictitle1">DWS Sink Stream (JDBC Mode)</h1>
|
|
<div id="body1574393095758"><div class="section" id="dli_08_0247__en-us_topic_0117559458_section848675393917"><h4 class="sectiontitle">Function</h4><p id="dli_08_0247__en-us_topic_0117559458_p69241927131319">DLI outputs the Flink job output data to Data Warehouse Service (DWS). DWS database kernel is compliant with PostgreSQL. The PostgreSQL database can store data of more complex types and delivers space information services, multi-version concurrent control (MVCC), and high concurrency. It applies to location applications, financial insurance, and e-commerce.</p>
|
|
<p id="dli_08_0247__en-us_topic_0117559458_p163354434173">DWS is an online data processing database based on the cloud infrastructure and platform and helps you mine and analyze massive sets of data. For more information about DWS, see the <em id="dli_08_0247__i659014584616"></em>.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0247__en-us_topic_0117559458_section1773605853811"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0247__en-us_topic_0117559458_ul810551215398"><li id="dli_08_0247__en-us_topic_0117559458_li6425436163814">Ensure that you have created a DWS cluster on DWS using your account.<p id="dli_08_0247__en-us_topic_0117559458_p823517563510"><a name="dli_08_0247__en-us_topic_0117559458_li6425436163814"></a><a name="en-us_topic_0117559458_li6425436163814"></a>For details about how to create a DWS cluster, see <span class="filepath" id="dli_08_0247__en-us_topic_0117559458_filepath1232123081516"><b>Creating a Cluster</b></span> in the <em id="dli_08_0247__en-us_topic_0117559458_i182321530161518">Data Warehouse Service Management Guide</em>.</p>
|
|
</li><li id="dli_08_0247__en-us_topic_0117559458_li240420488342">Ensure that a DWS database table has been created.</li><li id="dli_08_0247__en-us_topic_0117559458_li14111191212396">In this scenario, jobs must run on the dedicated queue of DLI. Therefore, DLI must interconnect with the enhanced datasource connection that has been connected with DWS clusters. You can also set the security group rules as required.<p id="dli_08_0247__p94091618103916"><a name="dli_08_0247__en-us_topic_0117559458_li14111191212396"></a><a name="en-us_topic_0117559458_li14111191212396"></a>For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_08_0247__dli_08_0249_en-us_topic_0132966325_filepath13726182511815"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_08_0247__dli_08_0249_cite534516e564224b13bad1de4471da3139181527">Data Lake Insight User Guide</cite></i>.</p>
|
|
<p id="dli_08_0247__p53271612101619">For details about how to configure security group rules, see <span class="filepath" id="dli_08_0247__dli_08_0249_filepath78235157113"><b>Security Group</b></span> in the <em id="dli_08_0247__dli_08_0249_i082519153111">Virtual Private Cloud User Guide</em>.</p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0247__en-us_topic_0117559458_section5292631103113"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0247__screen17988123520492"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span>
|
|
<span class="normal">8</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">stream_id</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"rds"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">db_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span>
|
|
<span class="w"> </span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0247__section1821218561501"><h4 class="sectiontitle">Keywords</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0247__en-us_topic_0117559458_table1648420306385" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0247__en-us_topic_0117559458_row248543016387"><th align="left" class="cellrowborder" valign="top" width="11.53%" id="mcps1.3.4.2.2.4.1.1"><p id="dli_08_0247__en-us_topic_0117559458_p3485130143810">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="9.87%" id="mcps1.3.4.2.2.4.1.2"><p id="dli_08_0247__en-us_topic_0117559458_p168561350102814">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="78.60000000000001%" id="mcps1.3.4.2.2.4.1.3"><p id="dli_08_0247__en-us_topic_0117559458_p148513023812">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0247__en-us_topic_0117559458_row19485193073810"><td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0247__en-us_topic_0117559458_p3485330133817">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.87%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0247__en-us_topic_0117559458_p08567508283">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.60000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0247__en-us_topic_0117559458_p13485123063812">Output channel type. <strong id="dli_08_0247__en-us_topic_0117559458_b842352706193629">rds</strong> indicates that data is exported to RDS or DWS.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0247__en-us_topic_0117559458_row9485143010387"><td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0247__en-us_topic_0117559458_p1748515305387">username</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.87%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0247__en-us_topic_0117559458_p1385665012811">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.60000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0247__en-us_topic_0117559458_p548513301385">Username for connecting to a database.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0247__en-us_topic_0117559458_row6485163083810"><td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0247__en-us_topic_0117559458_p1248511308389">password</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.87%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0247__en-us_topic_0117559458_p78566504284">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.60000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0247__en-us_topic_0117559458_p148573019388">Password for connecting to a database.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0247__en-us_topic_0117559458_row648533013820"><td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0247__en-us_topic_0117559458_p248510306381">db_url</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.87%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0247__en-us_topic_0117559458_p10856145082816">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.60000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0247__en-us_topic_0117559458_p9758112134510">Database connection address, for example, <strong id="dli_08_0247__b132111278477">postgresql://ip:port/database</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0247__en-us_topic_0117559458_row2661452184316"><td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0247__en-us_topic_0117559458_p18662052154317">table_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.87%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0247__en-us_topic_0117559458_p1585655032811">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.60000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0247__en-us_topic_0117559458_p1566312520436">Name of the table where data will be inserted. You need to create the database table in advance.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0247__en-us_topic_0117559458_row123510295489"><td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0247__en-us_topic_0117559458_p11506104112011">db_columns</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.87%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0247__en-us_topic_0117559458_p95068492017">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.60000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0247__en-us_topic_0117559458_p95061743206">Mapping between attributes in the output stream and those in the database table. This parameter must be configured based on the sequence of attributes in the output stream.</p>
|
|
<p id="dli_08_0247__en-us_topic_0117559458_p165848247233">Example:</p>
|
|
<div class="codecoloring" codetype="Sql" id="dli_08_0247__en-us_topic_0117559458_screen28841856122619"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span>
|
|
<span class="normal">8</span>
|
|
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="n">sink</span><span class="w"> </span><span class="n">stream</span><span class="w"> </span><span class="n">a3</span><span class="p">(</span><span class="n">student_name</span><span class="w"> </span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">student_age</span><span class="w"> </span><span class="nb">int</span><span class="p">)</span><span class="w"> </span>
|
|
<span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"rds"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"root"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"xxxxxxxx"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">db_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"postgresql://192.168.0.102:8000/test1"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">db_columns</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"name,age"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"t1"</span>
|
|
<span class="w"> </span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
<p id="dli_08_0247__en-us_topic_0117559458_p19798110172017">In the example, <strong id="dli_08_0247__en-us_topic_0117559458_b8423527062051">student_name</strong> corresponds to the name attribute in the database, and <strong id="dli_08_0247__en-us_topic_0117559458_b84235270620512">student_age</strong> corresponds to the age attribute in the database.</p>
|
|
<div class="note" id="dli_08_0247__en-us_topic_0117559458_note187991037132716"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0247__ul436117539341"><li id="dli_08_0247__li9361155319341">If <strong id="dli_08_0247__en-us_topic_0117559458_b84235270620532">db_columns</strong> is not configured, it is normal that the number of attributes in the output stream is less than that of attributes in the database table and the extra attributes in the database table are all nullable or have default values.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0247__en-us_topic_0117559458_row6584144623319"><td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0247__en-us_topic_0117559458_p5917239431">primary_key</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.87%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0247__en-us_topic_0117559458_p14912236436">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="78.60000000000001%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0247__en-us_topic_0117559458_p1251143121719">To update data in the table in real time by using the primary key, add the <strong id="dli_08_0247__en-us_topic_0117559458_b84235270616106">primary_key</strong> configuration item (<strong id="dli_08_0247__en-us_topic_0117559458_b2042108653161034">c_timeminute</strong> in the following example) when creating a table. During the data writing operation, data is updated if the specified <strong id="dli_08_0247__en-us_topic_0117559458_b84235270616118">primary_key</strong> exists. Otherwise, data is inserted.</p>
|
|
<p id="dli_08_0247__en-us_topic_0117559458_p12866182174316">Example:</p>
|
|
<div class="codecoloring" codetype="Sql" id="dli_08_0247__en-us_topic_0117559458_screen15277181717477"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span>
|
|
<span class="normal">8</span>
|
|
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">test</span><span class="p">(</span><span class="n">c_timeminute</span><span class="w"> </span><span class="n">LONG</span><span class="p">,</span><span class="w"> </span><span class="n">c_cnt</span><span class="w"> </span><span class="n">LONG</span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"rds"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"root"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"xxxxxxxx"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">db_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"postgresql://192.168.0.12:8000/test"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"test"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">primary_key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"c_timeminute"</span>
|
|
<span class="w"> </span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0247__section02441912175313"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0247__en-us_topic_0117559458_p13386336154418">The stream format defined by <strong id="dli_08_0247__en-us_topic_0117559458_b842352706193745">stream_id</strong> must be the same as the table format.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0247__en-us_topic_0117559458_section171023793119"><h4 class="sectiontitle">Example</h4><p id="dli_08_0247__en-us_topic_0117559458_p12360185814444">Data of stream <strong id="dli_08_0247__en-us_topic_0117559458_b84235270614114">audi_cheaper_than_30w</strong> is exported to the <strong id="dli_08_0247__en-us_topic_0117559458_b84235270694052">audi_cheaper_than_30w</strong> table in the <strong id="dli_08_0247__en-us_topic_0117559458_b842352706141114">test</strong> database.</p>
|
|
<div class="codecoloring" codetype="Sql" id="dli_08_0247__en-us_topic_0117559458_screen15458151114815"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">audi_cheaper_than_30w</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_price</span><span class="w"> </span><span class="nb">INT</span>
|
|
<span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"rds"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"root"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"xxxxxx"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">db_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"postgresql://192.168.1.1:8000/test"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"audi_cheaper_than_30w"</span>
|
|
<span class="w"> </span><span class="p">);</span><span class="w"> </span>
|
|
|
|
<span class="k">insert</span><span class="w"> </span><span class="k">into</span><span class="w"> </span><span class="n">audi_cheaper_than_30w</span><span class="w"> </span><span class="k">select</span><span class="w"> </span><span class="ss">"1"</span><span class="p">,</span><span class="ss">"2"</span><span class="p">,</span><span class="ss">"3"</span><span class="p">,</span><span class="mi">4</span><span class="p">;</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0240.html">Creating a Sink Stream</a></div>
|
|
</div>
|
|
</div>
|
|
|