vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0394.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

329 lines
40 KiB
HTML
Raw Blame History

<a name="dli_08_0394"></a><a name="dli_08_0394"></a>
<h1 class="topictitle1">GaussDB(DWS) Result Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_section108631122164917"><h4 class="sectiontitle">Function</h4><p id="dli_08_0394__en-us_topic_0000001310095785_p161343396413">DLI outputs the Flink job output data to GaussDB(DWS). GaussDB(DWS) database kernel is compliant with PostgreSQL. The PostgreSQL database can store data of more complex types and deliver space information services, multi-version concurrent control (MVCC), and high concurrency. It applies to location applications, financial insurance, and e-Commerce.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p113419391416">GaussDB(DWS) is an online data processing database based on the cloud infrastructure and platform and helps you mine and analyze massive sets of data. </p>
</div>
<div class="section" id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_section2579142713429"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0394__en-us_topic_0000001310095785_ul29504272194"><li id="dli_08_0394__en-us_topic_0000001310095785_li134871014122418">When creating a Flink OpenSource SQL job, you need to set <strong id="dli_08_0394__en-us_topic_0000001310095785_b96017211601">Flink Version</strong> to <strong id="dli_08_0394__en-us_topic_0000001310095785_b176162117018">1.12</strong> on the <strong id="dli_08_0394__en-us_topic_0000001310095785_b206112211108">Running Parameters</strong> tab of the job editing page, select <strong id="dli_08_0394__en-us_topic_0000001310095785_b116222119012">Save Job Log</strong>, and set the OBS bucket for saving job logs.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li6950202711198">You have created a GaussDB(DWS) cluster. For details about how to create a GaussDB(DWS) cluster, see <span class="filepath" id="dli_08_0394__en-us_topic_0000001310095785_filepath677311615318"><b>Creating a Cluster</b></span> in the <em id="dli_08_0394__en-us_topic_0000001310095785_i9773131612320">Data Warehouse Service Management Guide</em>.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li149509273195">You have created a GaussDB(DWS) database table.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li495011273191">An enhanced datasource connection has been created for DLI to connect to GaussDB(DWS) clusters, so that jobs can run on the dedicated queue of DLI and you can set the security group rules as required.
</li></ul>
</div>
<div class="section" id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_section1485175513219"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0394__en-us_topic_0000001310095785_ul078064616449"><li id="dli_08_0394__en-us_topic_0000001310095785_li68011626122214">To use the upsert mode, you must define the primary key for both the GaussDB(DWS) result table and the GaussDB(DWS) table connected to the result table.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li186672521377">If tables with the same name exist in different GaussDB(DWS) schemas, you need to specify the schemas in the Flink open source SQL statements.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li181413382114">Before submitting a Flink job, you are advised to select <strong id="dli_08_0394__en-us_topic_0000001310095785_b381119339365">Save Job Log</strong> and set the OBS bucket for saving job logs. This helps you view logs and locate faults when the job fails to be submitted or runs abnormally.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li11543332172518">If you use the gsjdbc4 driver for connection, set <strong id="dli_08_0394__en-us_topic_0000001310095785_b17412638153915">driver</strong> to <strong id="dli_08_0394__en-us_topic_0000001310095785_b154190388391">org.postgresql.Driver</strong>. You can omit this parameter because the gsjdbc4 driver is the default one.<div class="p" id="dli_08_0394__en-us_topic_0000001310095785_p119313128239">For example, run the following statements to use the gsjdbc4 driver to write data to GaussDB(DWS) in upsert mode:<div class="codecoloring" codetype="Sql" id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_screen267912844511"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">dwsSink</span><span class="p">(</span>
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_speed</span><span class="w"> </span><span class="nb">INT</span>
<span class="p">)</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'gaussdb'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'url'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'jdbc:postgresql://DwsAddress:DwsPort/DwsDatabase'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'table-name'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'car_info'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'username'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'DwsUserName'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'password'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'DwsPasswrod'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'write.mode'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'upsert'</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
</li></ul>
</div>
<div class="section" id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_section3126105364419"><h4 class="sectiontitle">Syntax</h4><div class="note" id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001127915813_note12145122711222"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001127915813_p4210132818222">Do not set all attributes in a GaussDB(DWS) result table to <strong id="dli_08_0394__en-us_topic_0000001310095785_b756164573115">PRIMARY KEY</strong>.</p>
</div></div>
<div class="codecoloring" codetype="Sql" id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0243_screen746013311958"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">dwsSink</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="p">,</span><span class="w"> </span><span class="p">...)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">ENFORCED</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">with</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="s1">'connector'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'gaussdb'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'url'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'table-name'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'driver'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'username'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'password'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_section538311333216"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_table73321814102516" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_row1934731413256"><th align="left" class="cellrowborder" valign="top" width="17.57175717571757%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_p163529142259">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.53095309530953%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_p17357514162518">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="11.03110311031103%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0394__en-us_topic_0000001310095785_p1439841382614">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.14091409140914%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0394__en-us_topic_0000001310095785_p122782112616">Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="52.72527252725272%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_p936111422510">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_row636719141253"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_p16266123442318">connector</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_p142661034102314">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p739881320264">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p10271214262">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1526613419233">Connector to be used. Set this parameter to <strong id="dli_08_0394__en-us_topic_0000001310095785_b9246105835516">gaussdb</strong>.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_row17383101422514"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1743276019">url</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1474427901">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p10398101392614">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p0273213269">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_p8869151717249">JDBC connection address.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p17701919201416">If you use the gsjdbc4 driver, set the value in jdbc:postgresql://${ip}:${port}/${dbName} format.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p87442714011">If you use the gsjdbc200 driver, set the value in jdbc:gaussdb://${ip}:${port}/${dbName} format.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_row9401214162511"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_p926610342236">table-name</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_p192664348233">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p16316115821612">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1127112122616">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_p8122151404417">Name of the table to be operated. If the GaussDB(DWS) table is in a schema, the format is <strong id="dli_08_0394__en-us_topic_0000001310095785_b17547174219566">schema\".\"</strong><em id="dli_08_0394__en-us_topic_0000001310095785_i1255374212566">Table name</em>. For details, see <a href="#dli_08_0394__en-us_topic_0000001310095785_section92924310151">FAQ</a>.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_row1141921417251"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_p2266173415231">driver</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_p92669342234">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p11398101315265">org.postgresql.Driver</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p14278214262">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p1548125111205">JDBC connection driver. The default value is <strong id="dli_08_0394__en-us_topic_0000001310095785_b121303613576">org.postgresql.Driver</strong>.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_row1643713148256"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_p8266113414235">username</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1326653418238">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p193986131264">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1927132172620">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1726613410230">Username for GaussDB(DWS) database authentication. This parameter must be configured in pair with <strong id="dli_08_0394__en-us_topic_0000001310095785_b1735582591">password</strong>.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_en-us_topic_0111499976_row134850141257"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_p152661434112310">password</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_p122667341236">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p01681827179">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p14271021102616">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_p14266134122317">Password for GaussDB(DWS) database authentication. This parameter must be configured in pair with <strong id="dli_08_0394__en-us_topic_0000001310095785_b14178857017">username</strong>.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_row2072024516365"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p380117914352">write.mode</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p1680139163512">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1139921342618">None</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p927321122619">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_p16145125214206">Data write mode. The value can be <strong id="dli_08_0394__en-us_topic_0000001310095785_b192941111628">copy</strong>, <strong id="dli_08_0394__en-us_topic_0000001310095785_b330114119219">insert</strong>, or <strong id="dli_08_0394__en-us_topic_0000001310095785_b83026111921">upsert</strong>. The default value is <strong id="dli_08_0394__en-us_topic_0000001310095785_b122804272345330">upsert</strong>.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p553424021914">This parameter must be configured depending on <strong id="dli_08_0394__en-us_topic_0000001310095785_b187319181128">primary key</strong>.</p>
<ul id="dli_08_0394__en-us_topic_0000001310095785_ul394174718190"><li id="dli_08_0394__en-us_topic_0000001310095785_li39414475197">If <strong id="dli_08_0394__en-us_topic_0000001310095785_b260673919213">primary key</strong> is not configured, data can be appended in <strong id="dli_08_0394__en-us_topic_0000001310095785_b7607239829">copy</strong> and <strong id="dli_08_0394__en-us_topic_0000001310095785_b166074391722">insert</strong> modes.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li25391325216">If <strong id="dli_08_0394__en-us_topic_0000001310095785_b044918491425">primary key</strong> is configured, all the three modes are available.</li></ul>
<p id="dli_08_0394__en-us_topic_0000001310095785_p14591743213">Note: GaussDB(DWS) does not support the update of distribution columns. The primary keys of columns to be updated must cover all distribution columns defined in the GaussDB(DWS) table.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_row11633185213369"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p580279103516">sink.buffer-flush.max-rows</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p1180214983519">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p639951352613">100</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p162742122618">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_p57071536393">Maximum number of rows to buffer for each write request.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p16707731399">It can improve the performance of writing data, but may increase the latency.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p15707639399">You can set this parameter to <strong id="dli_08_0394__en-us_topic_0000001310095785_b334534712266">0</strong> to disable it.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_row1934512550360"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p1080211912353">sink.buffer-flush.interval</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p18024913359">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p93991613192613">1s</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1927921202615">Duration</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1552693154019">Interval for refreshing the buffer, during which data is refreshed by asynchronous threads.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p1552613194013">It can improve the performance of writing data to the database, but may increase the latency.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p4526183184013">You can set this parameter to <strong id="dli_08_0394__en-us_topic_0000001310095785_b479514478271">0</strong> to disable it.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p15261237404">Note: If <strong id="dli_08_0394__en-us_topic_0000001310095785_b13199612112810">sink.buffer-flush.max-size</strong> and <strong id="dli_08_0394__en-us_topic_0000001310095785_b31175186280">sink.buffer-flush.max-rows</strong> are both set to <strong id="dli_08_0394__en-us_topic_0000001310095785_b1949194152810">0</strong> and the buffer refresh interval is configured, the buffer is asynchronously refreshed.</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p2063564452519">The format is {length value}{time unit label}, for example, <strong id="dli_08_0394__en-us_topic_0000001310095785_b14412173413122">123ms, 321s</strong>. The supported time units include d, h, min, s, and ms (default unit).</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_row168857116372"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p3802995350">sink.max-retries</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p208031491353">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1639941316266">3</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1827122120265">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p138031593358">Maximum number of write retries.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_row103573912371"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p617632444319">write.escape-string-value</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p617692484320">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p113991113192619">false</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p19271021102611">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_en-us_topic_0000001080679435_p181761124164318">Whether to escape values of the string type. This parameter is used only when <strong id="dli_08_0394__en-us_topic_0000001310095785_b9331203911313">write.mode</strong> is set to <strong id="dli_08_0394__en-us_topic_0000001310095785_b198754403132">copy</strong>.</p>
</td>
</tr>
<tr id="dli_08_0394__en-us_topic_0000001310095785_row11347820351"><td class="cellrowborder" valign="top" width="17.57175717571757%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0394__en-us_topic_0000001310095785_p034762014518">key-by-before-sink</p>
</td>
<td class="cellrowborder" valign="top" width="9.53095309530953%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0394__en-us_topic_0000001310095785_p23471201754">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.03110311031103%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1334716201150">false</p>
</td>
<td class="cellrowborder" valign="top" width="9.14091409140914%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0394__en-us_topic_0000001310095785_p53481206513">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="52.72527252725272%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0394__en-us_topic_0000001310095785_p1758218468520">Whether to partition by the specified primary key before the sink operator</p>
<p id="dli_08_0394__en-us_topic_0000001310095785_p1058213464513">This parameter aims to solve the problem of interlocking between two subtasks when they acquire row locks based on the primary key from GaussDB(DWS), multiple concurrent writes occur, and <strong id="dli_08_0394__en-us_topic_0000001310095785_b3603132275419">write.mode</strong> is <strong id="dli_08_0394__en-us_topic_0000001310095785_b48094231546">upsert</strong>. This happens when a batch of data written to the sink by multiple subtasks has more than one record with the same primary key, and the order of these records with the same primary key is inconsistent.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0394__en-us_topic_0000001310095785_dli_08_0252_section085022413338"><h4 class="sectiontitle">Example</h4><p id="dli_08_0394__en-us_topic_0000001310095785_p11305132402910">In this example, data is read from the Kafka data source and written to the GaussDB(DWS) result table in insert mode. The procedure is as follows:</p>
<ol id="dli_08_0394__en-us_topic_0000001310095785_ol1993816019209"><li id="dli_08_0394__en-us_topic_0000001310095785_li6127741153217">Create an enhanced datasource connection in the VPC and subnet where GaussDB(DWS) and Kafka locate, and bind the connection to the required Flink elastic resource pool.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li1233002211514">Set GaussDB(DWS) and Kafka security groups and add inbound rules to allow access from the Flink queue. Test the connectivity using the GaussDB(DWS) and Kafka address. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li139381100208">Connect to the GaussDB(DWS) database and create a table named <strong id="dli_08_0394__en-us_topic_0000001310095785_b1692451371717">dws_order</strong>.<pre class="screen" id="dli_08_0394__en-us_topic_0000001310095785_screen1510764661915">create table public.dws_order(
order_id VARCHAR,
order_channel VARCHAR,
order_time VARCHAR,
pay_amount FLOAT8,
real_pay FLOAT8,
pay_time VARCHAR,
user_id VARCHAR,
user_name VARCHAR,
area_id VARCHAR);</pre>
</li><li id="dli_08_0394__en-us_topic_0000001310095785_li10938707209">Create a Flink OpenSource SQL job. Enter the following job script and submit the job. The job script uses the Kafka data source and the GaussDB(DWS) result table.<div class="p" id="dli_08_0394__en-us_topic_0000001310095785_p623882218414"><a name="dli_08_0394__en-us_topic_0000001310095785_li10938707209"></a><a name="en-us_topic_0000001310095785_li10938707209"></a>When you create a job, set <strong id="dli_08_0394__en-us_topic_0000001310095785_b3968181110575">Flink Version</strong> to <strong id="dli_08_0394__en-us_topic_0000001310095785_b12968171135711">1.12</strong> on the <strong id="dli_08_0394__en-us_topic_0000001310095785_b119698119571">Running Parameters</strong> tab. Select <strong id="dli_08_0394__en-us_topic_0000001310095785_b15969121113575">Save Job Log</strong>, and specify the OBS bucket for saving job logs. <strong id="dli_08_0394__en-us_topic_0000001310095785_b274651355712">Change the values of the parameters in bold as needed in the following script.</strong><pre class="screen" id="dli_08_0394__en-us_topic_0000001310095785_screen166321446193113">CREATE TABLE kafkaSource (
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = '<em id="dli_08_0394__en-us_topic_0000001310095785_i4543155265320"><strong id="dli_08_0394__en-us_topic_0000001310095785_b1754316528535">Kafka</strong></em><em id="dli_08_0394__en-us_topic_0000001310095785_i15341759195319"><strong id="dli_08_0394__en-us_topic_0000001310095785_b5227105913538">Topic</strong></em>',
'properties.bootstrap.servers' = '<em id="dli_08_0394__en-us_topic_0000001310095785_i9319103215313"><strong id="dli_08_0394__en-us_topic_0000001310095785_b0322115133213">KafkaAddress1:KafkaPort,KafkaAddress2:KafkaPort</strong></em>',
'properties.group.id' = '<em id="dli_08_0394__en-us_topic_0000001310095785_i14535174011311"><strong id="dli_08_0394__en-us_topic_0000001310095785_b1466610178329">GroupId</strong></em>',
'scan.startup.mode' = 'latest-offset',
'format' = 'json'
);
CREATE TABLE dwsSink (
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'gaussdb',
'url' = 'jdbc:postgresql://<em id="dli_08_0394__en-us_topic_0000001310095785_i2068813014544"><strong id="dli_08_0394__en-us_topic_0000001310095785_b6427143015414">DWSAddress:DWSPort</strong></em>/<em id="dli_08_0394__en-us_topic_0000001310095785_i3295722133411"><strong id="dli_08_0394__en-us_topic_0000001310095785_b6280224349">DWSdbName</strong></em>',
'table-name' = 'dws_order',
'driver' = 'org.postgresql.Driver',
'username' = '<em id="dli_08_0394__en-us_topic_0000001310095785_i193698287344"><strong id="dli_08_0394__en-us_topic_0000001310095785_b163691228163419">DWS</strong></em><em id="dli_08_0394__en-us_topic_0000001310095785_i145941322341"><strong id="dli_08_0394__en-us_topic_0000001310095785_b126416322341">UserName</strong></em>',
'password' = '<em id="dli_08_0394__en-us_topic_0000001310095785_i11414537113414"><strong id="dli_08_0394__en-us_topic_0000001310095785_b84149375345">DWS</strong></em><em id="dli_08_0394__en-us_topic_0000001310095785_i6532114116345"><strong id="dli_08_0394__en-us_topic_0000001310095785_b82261641193414">Password</strong></em>',
'write.mode' = 'insert'
);
insert into dwsSink select * from kafkaSource;</pre>
</div>
</li><li id="dli_08_0394__en-us_topic_0000001310095785_li1193980192011">Connect to the Kafka cluster and enter the following test data to Kafka:<pre class="screen" id="dli_08_0394__en-us_topic_0000001310095785_screen88321672428">{"order_id":"202103241000000001", "order_channel":"webShop", "order_time":"2021-03-24 10:00:00", "pay_amount":"100.00", "real_pay":"100.00", "pay_time":"2021-03-24 10:02:03", "user_id":"0001", "user_name":"Alice", "area_id":"330106"}</pre>
</li><li id="dli_08_0394__en-us_topic_0000001310095785_li1793912013207">Run the following SQL statement in GaussDB(DWS) to view the data result:<pre class="screen" id="dli_08_0394__en-us_topic_0000001310095785_screen2036145011442"> select * from dws_order</pre>
<div class="p" id="dli_08_0394__en-us_topic_0000001310095785_p13880930183520">The data result is as follows:<pre class="screen" id="dli_08_0394__en-us_topic_0000001310095785_screen651832924515">202103241000000001 webShop 2021-03-24 10:00:00 100.0 100.0 2021-03-24 10:02:03 0001 Alice 330106</pre>
</div>
</li></ol>
</div>
<div class="section" id="dli_08_0394__en-us_topic_0000001310095785_section92924310151"><a name="dli_08_0394__en-us_topic_0000001310095785_section92924310151"></a><a name="en-us_topic_0000001310095785_section92924310151"></a><h4 class="sectiontitle">FAQ</h4><ul id="dli_08_0394__en-us_topic_0000001310095785_ul141114560153"><li id="dli_08_0394__en-us_topic_0000001310095785_li1263872453620">Q: What should I do if the Flink job execution fails and the log contains the following error information?<pre class="screen" id="dli_08_0394__en-us_topic_0000001310095785_screen16117181494">java.io.IOException: unable to open JDBC writer
...
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
...
Caused by: java.net.SocketTimeoutException: connect timed out</pre>
<div class="p" id="dli_08_0394__en-us_topic_0000001310095785_p116875264365">A: The datasource connection is not bound or the binding fails.
</div>
</li></ul>
<ul id="dli_08_0394__en-us_topic_0000001310095785_ul2033515489478"><li id="dli_08_0394__en-us_topic_0000001310095785_li7335184854719">Q: How can I configure a GaussDB(DWS) table that is in a schema?<div class="p" id="dli_08_0394__en-us_topic_0000001310095785_p969020584512"><a name="dli_08_0394__en-us_topic_0000001310095785_li7335184854719"></a><a name="en-us_topic_0000001310095785_li7335184854719"></a>A: When GaussDB(DWS) table <strong id="dli_08_0394__en-us_topic_0000001310095785_b1599311791215">test</strong> is in schema <strong id="dli_08_0394__en-us_topic_0000001310095785_b1029652311126">ads_game_sdk_base</strong>, refer to the <strong id="dli_08_0394__en-us_topic_0000001310095785_b2444317123">'table-name'</strong> parameter setting in the following example:<pre class="screen" id="dli_08_0394__en-us_topic_0000001310095785_screen89157571494">CREATE TABLE ads_rpt_game_sdk_realtime_ada_reg_user_pay_mm (
ddate DATE,
dmin TIMESTAMP(3),
game_appkey VARCHAR,
channel_id VARCHAR,
pay_user_num_1m bigint,
pay_amt_1m bigint,
PRIMARY KEY (ddate, dmin, game_appkey, channel_id) NOT ENFORCED
) WITH (
'connector' = 'gaussdb',
'url' = 'jdbc:postgresql://&lt;yourDwsAddress&gt;:&lt;yourDwsPort&gt;/dws_bigdata_db',
'table-name' = <strong id="dli_08_0394__en-us_topic_0000001310095785_b16914195715493">'ads_game_sdk_base\".\"test'</strong>,
'username' = '&lt;yourUsername&gt;',
'password' = '&lt;yourPassword&gt;',
'write.mode' = 'upsert'
);</pre>
</div>
</li></ul>
</div>
<ul id="dli_08_0394__en-us_topic_0000001310095785_ul151451826135710"><li id="dli_08_0394__en-us_topic_0000001310095785_li4145202613574">Q: What can I do if a job is running properly but there is no data in GaussDB(DWS)?<div class="p" id="dli_08_0394__en-us_topic_0000001310095785_p880244575714"><a name="dli_08_0394__en-us_topic_0000001310095785_li4145202613574"></a><a name="en-us_topic_0000001310095785_li4145202613574"></a>A: Check the following items:<ul id="dli_08_0394__en-us_topic_0000001310095785_ul3972134495710"><li id="dli_08_0394__en-us_topic_0000001310095785_li16972844135719">Check whether the JobManager and TaskManager logs contain error information. To view logs, perform the following steps:<ol id="dli_08_0394__en-us_topic_0000001310095785_ol109721447574"><li id="dli_08_0394__en-us_topic_0000001310095785_li1497264435716">Log in to the DLI console. In the navigation pane, choose <strong id="dli_08_0394__en-us_topic_0000001310095785_b13356184071615">Job Management</strong> &gt; <strong id="dli_08_0394__en-us_topic_0000001310095785_b1135734017167">Flink Jobs</strong>.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li3972124475720">Click the name of the corresponding Flink job, choose <strong id="dli_08_0394__en-us_topic_0000001310095785_b12152194671615">Run Log</strong>, click <strong id="dli_08_0394__en-us_topic_0000001310095785_b115884615168">OBS Bucket</strong>, and locate the folder of the log you want to view according to the date.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li8972204410579">Go to the folder of the date, find the folder whose name contains <strong id="dli_08_0394__en-us_topic_0000001310095785_b11385105021613">taskmanager</strong> or <strong id="dli_08_0394__en-us_topic_0000001310095785_b127911112111717">jobmanager</strong>, download the <strong id="dli_08_0394__en-us_topic_0000001310095785_b1738555014163">taskmanager.out</strong> or <strong id="dli_08_0394__en-us_topic_0000001310095785_b14165122612171">jobmanager.out</strong> file, and view result logs.</li></ol>
</li><li id="dli_08_0394__en-us_topic_0000001310095785_li89725442578">Check whether the datasource connection is correctly bound and whether a security group rule allows access of the queue.</li><li id="dli_08_0394__en-us_topic_0000001310095785_li11972154416576">Check whether the GaussDB(DWS) table to which data is to be written exists in multiple schemas. If it does, specify the schemas in the Flink job.</li></ul>
</div>
</li></ul>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0391.html">Creating Result Tables</a></div>
</div>
</div>
View Git Blame Copy Permalink