vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0248.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

291 lines
32 KiB
HTML
Raw Blame History

<a name="dli_08_0248"></a><a name="dli_08_0248"></a>
<h1 class="topictitle1">DWS Sink Stream (OBS-based Dumping)</h1>
<div id="body1574393100356"><div class="section" id="dli_08_0248__en-us_topic_0125566718_section848675393917"><h4 class="sectiontitle">Function</h4><p id="dli_08_0248__en-us_topic_0125566718_p025717281016">Create a sink stream to export Flink job data to DWS through OBS-based dumping, specifically, output Flink job data to OBS and then import data from OBS to DWS. For details about how to import OBS data to DWS, see <span class="filepath" id="dli_08_0248__en-us_topic_0125566718_filepath175951081716"><b>Concurrently Importing Data from OBS</b></span> in the Data Warehouse Service Development Guide<em id="dli_08_0248__en-us_topic_0125566718_i0802524151120">Data Warehouse Service Development Guide</em>.</p>
<p id="dli_08_0248__en-us_topic_0125566718_p163354434173">DWS is an online data processing database based on the cloud infrastructure and platform and helps you mine and analyze massive sets of data. For more information about DWS, see the <em id="dli_08_0248__dli_08_0247_i659014584616"></em>.</p>
</div>
<div class="section" id="dli_08_0248__en-us_topic_0125566718_section5426161831913"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0248__en-us_topic_0125566718_ul68278459198"><li id="dli_08_0248__en-us_topic_0125566718_li78271945141916">OBS-based dumping supports intermediate files of the following two types:<ul id="dli_08_0248__en-us_topic_0125566718_ul3252125217180"><li id="dli_08_0248__en-us_topic_0125566718_li169963072010">ORC: The ORC format does not support array data type. If the ORC format is used, create a foreign server in DWS. For details, see <span class="filepath" id="dli_08_0248__filepath1442581984713"><b>Creating a Foreign Server</b></span> in the <em id="dli_08_0248__i2572759132412"></em>.</li><li id="dli_08_0248__en-us_topic_0125566718_li72521352161811">CSV: By default, the line break is used as the record separator. If the line break is contained in the attribute content, you are advised to configure quote. For details, see <a href="#dli_08_0248__en-us_topic_0125566718_table1648420306385">Table 1</a>.</li></ul>
</li><li id="dli_08_0248__en-us_topic_0125566718_li378419082011">If the target table does not exist, a table is automatically created. DLI data of the SQL type does not support <strong id="dli_08_0248__b11716143513311">text</strong>. If a long text exists, you are advised to create a table in the database.</li><li id="dli_08_0248__li965712334111">When <strong id="dli_08_0248__b2171941823">encode</strong> uses the ORC format to create a DWS table, if the field attribute of the SQL stream is defined as the <strong id="dli_08_0248__b177111191620">String</strong> type, the field attribute of the DWS table cannot use the <strong id="dli_08_0248__b1156682917217">varchar</strong> type. Instead, a specific text type must be used. If the SQL stream field attribute is defined as the <strong id="dli_08_0248__b478218518216">Integer</strong> type, the DWS table field must use the <strong id="dli_08_0248__b11331658026">Integer</strong> type.</li></ul>
</div>
<div class="section" id="dli_08_0248__en-us_topic_0125566718_section1263773412592"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0248__en-us_topic_0125566718_ul6321351125913"><li id="dli_08_0248__en-us_topic_0125566718_li174951814134919">Ensure that OBS buckets and folders have been created.<p id="dli_08_0248__en-us_topic_0125566718_p10965155831114"><a name="dli_08_0248__en-us_topic_0125566718_li174951814134919"></a><a name="en-us_topic_0125566718_li174951814134919"></a>For details about how to create an OBS bucket, see <span class="filepath" id="dli_08_0248__en-us_topic_0125566718_filepath16716312195"><b>Creating a Bucket</b></span> in the <em id="dli_08_0248__en-us_topic_0125566718_i5198141916199">Object Storage Service User Guide</em>.</p>
<p id="dli_08_0248__en-us_topic_0125566718_p189706583119">For details about how to create a folder, see <span class="filepath" id="dli_08_0248__en-us_topic_0125566718_filepath122205238197"><b>Creating a Folder</b></span> in the <em id="dli_08_0248__en-us_topic_0125566718_i114381307195">Object Storage Service User Guide</em>.</p>
</li><li id="dli_08_0248__en-us_topic_0125566718_li14111191212396">In this scenario, jobs must run on the dedicated queue of DLI. Therefore, DLI must interconnect with the enhanced datasource connection that has been connected with DWS clusters. You can also set the security group rules as required.<p id="dli_08_0248__p94091618103916"><a name="dli_08_0248__en-us_topic_0125566718_li14111191212396"></a><a name="en-us_topic_0125566718_li14111191212396"></a>For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_08_0248__dli_08_0249_en-us_topic_0132966325_filepath13726182511815"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_08_0248__dli_08_0249_cite534516e564224b13bad1de4471da3139181527">Data Lake Insight User Guide</cite></i>.</p>
<p id="dli_08_0248__p2941220141619">For details about how to configure security group rules, see <span class="filepath" id="dli_08_0248__dli_08_0249_filepath78235157113"><b>Security Group</b></span> in the <em id="dli_08_0248__dli_08_0249_i082519153111">Virtual Private Cloud User Guide</em>.</p>
</li></ul>
</div>
<div class="section" id="dli_08_0248__en-us_topic_0125566718_section5292631103113"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0248__en-us_topic_0125566718_screen116951731102316"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">stream_id</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span><span class="p">)</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dws&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">ak</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">sk</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">field_delimiter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">quote</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">db_obs_server</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">obs_dir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">db_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">max_record_num_per_file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">dump_interval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span>
<span class="w"> </span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div class="section" id="dli_08_0248__section554814581541"><h4 class="sectiontitle">Keywords</h4>
<div class="tablenoborder"><a name="dli_08_0248__en-us_topic_0125566718_table1648420306385"></a><a name="en-us_topic_0125566718_table1648420306385"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0248__en-us_topic_0125566718_table1648420306385" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0248__en-us_topic_0125566718_row248543016387"><th align="left" class="cellrowborder" valign="top" width="20.64%" id="mcps1.3.5.2.2.4.1.1"><p id="dli_08_0248__en-us_topic_0125566718_p3485130143810">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="8.9%" id="mcps1.3.5.2.2.4.1.2"><p id="dli_08_0248__en-us_topic_0125566718_p168561350102814">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="70.46%" id="mcps1.3.5.2.2.4.1.3"><p id="dli_08_0248__en-us_topic_0125566718_p148513023812">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0248__en-us_topic_0125566718_row19485193073810"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p3485330133817">type</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p08567508283">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p13485123063812">Output channel type. <strong id="dli_08_0248__en-us_topic_0125566718_b842352706192948">dws</strong> indicates that data is exported to DWS.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row98143247345"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p1081452413412">region</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p881452403419">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p1981562463410">Region where DWS is located.</p>
</td>
</tr>
<tr id="dli_08_0248__row1389919133594"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__p111537416387">ak</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__p161531941193814">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__p141531741143816">Access Key ID (AK). </p>
</td>
</tr>
<tr id="dli_08_0248__row166691845916"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__p58994360384">sk</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__p20899163615388">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__p1989953683812">Secret access key used together with the ID of the AK. </p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row6749372259"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p97491676250">encode</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p47492073251">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p142254213250">Encoding format. Currently, CSV and ORC are supported.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row714014544218"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p4140245184211">field_delimiter</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p12140845104213">No</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p6140745164218">Separator used to separate every two attributes. This parameter needs to be configured if the CSV encoding mode is used. It is recommended that you use invisible characters as separators, for example, <strong id="dli_08_0248__en-us_topic_0125566718_b1698514204010">\u0006\u0002</strong>.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row544053810422"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p1444014388421">quote</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p64403385426">No</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p7440938134219">Single byte. It is recommended that invisible characters be used, for example, <strong id="dli_08_0248__en-us_topic_0125566718_b144411243014">u0007</strong>.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row161154155276"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p7115161511277">db_obs_server</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p1511571515275">No</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p111151515112717">Foreign server (for example, <strong id="dli_08_0248__en-us_topic_0125566718_b10971334151417">obs_server</strong>) that has been created in the database.</p>
<p id="dli_08_0248__en-us_topic_0125566718_p9914171813133">You need to specify this parameter if the ORC encoding mode is adopted.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row425981912714"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p1025931913275">obs_dir</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p1425941915274">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p72597193274">Directory for storing intermediate files. The directory is in the format of {Bucket name}/{Directory name}, for example, obs-a1/dir1/subdir.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row9485143010387"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p1748515305387">username</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p1385665012811">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p548513301385">Username for connecting to a database.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row6485163083810"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p1248511308389">password</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p78566504284">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p148573019388">Password for connecting to a database.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row648533013820"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p845313313273">db_url</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p745214352718">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p9758112134510">Database connection address. The format is /ip:port/database, for example, <strong id="dli_08_0248__en-us_topic_0125566718_b126619538411">192.168.1.21:8000/test1</strong>.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row2661452184316"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p16448738278">table_name</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p744433172718">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p14437315277">Data table name. If no table is available, a table is automatically created.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row123510295489"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p244223102711">max_record_num_per_file</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p1944173152710">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p154399332717">Maximum number of records that can be stored in a file. If the number of records in a file is less than the maximum value, the file will be dumped to OBS after one dumping period.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row15604914143312"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p18604414113311">dump_interval</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p116041814193319">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p76048142331">Dumping period. The unit is second.</p>
</td>
</tr>
<tr id="dli_08_0248__en-us_topic_0125566718_row926616446284"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p62671444152813">delete_obs_temp_file</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p16267204417283">No</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p16267544152816">Whether to delete temporary files on OBS. The default value is <strong id="dli_08_0248__b98640311147">true</strong>. If this parameter is set to <strong id="dli_08_0248__b148641231349">false</strong>, files on OBS will not be deleted. You need to manually clear the files.</p>
</td>
</tr>
<tr id="dli_08_0248__row165240810299"><td class="cellrowborder" valign="top" width="20.64%" headers="mcps1.3.5.2.2.4.1.1 "><p id="dli_08_0248__en-us_topic_0125566718_p439443122713">max_dump_file_num</p>
</td>
<td class="cellrowborder" valign="top" width="8.9%" headers="mcps1.3.5.2.2.4.1.2 "><p id="dli_08_0248__en-us_topic_0125566718_p153931310278">No</p>
</td>
<td class="cellrowborder" valign="top" width="70.46%" headers="mcps1.3.5.2.2.4.1.3 "><p id="dli_08_0248__en-us_topic_0125566718_p14391337273">Maximum number of files that can be dumped at a time. If the number of files to be dumped is less than the maximum value, the files will be dumped to OBS after one dumping period.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0248__en-us_topic_0125566718_section171023793119"><h4 class="sectiontitle">Example</h4><ul id="dli_08_0248__en-us_topic_0125566718_ul11951618393"><li id="dli_08_0248__en-us_topic_0125566718_li1719121610398">Dump files in CSV format.<div class="codecoloring" codetype="Sql" id="dli_08_0248__en-us_topic_0125566718_screen15458151114815"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">car_infos</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_price</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_timestamp</span><span class="w"> </span><span class="n">LONG</span>
<span class="p">)</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dws&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;xxx&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">ak</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">sk</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;csv&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">field_delimiter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;\u0006\u0006\u0002&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">quote</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;\u0007&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">obs_dir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dli-append-2/dws&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">db_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;192.168.1.12:8000/test1&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;table1&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">max_record_num_per_file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;100&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">dump_interval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;10&quot;</span>
<span class="w"> </span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_08_0248__en-us_topic_0125566718_li18492103811413">Dump files in ORC format.<div class="codecoloring" codetype="Sql" id="dli_08_0248__en-us_topic_0125566718_screen19435103020419"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">car_infos</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_price</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_timestamp</span><span class="w"> </span><span class="n">LONG</span>
<span class="p">)</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dws&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;xxx&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">ak</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">sk</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;orc&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">db_obs_server</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;obs_server&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">obs_dir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dli-append-2/dws&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">db_url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;192.168.1.12:8000/test1&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;table1&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">max_record_num_per_file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;100&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">dump_interval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;10&quot;</span>
<span class="w"> </span><span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0240.html">Creating a Sink Stream</a></div>
</div>
</div>
View Git Blame Copy Permalink