forked from docs/doc-exports
Su, Xiaomeng
76a5b1ee83
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
285 lines
34 KiB
HTML
285 lines
34 KiB
HTML
<a name="dli_08_0242"></a><a name="dli_08_0242"></a>
|
|
|
|
<h1 class="topictitle1">OBS Sink Stream</h1>
|
|
<div id="body1574393078783"><div class="section" id="dli_08_0242__en-us_topic_0125566402_section1826712033217"><h4 class="sectiontitle">Function</h4><p id="dli_08_0242__en-us_topic_0125566402_p2072222220463">Create a sink stream to export DLI data to OBS. DLI can export the job analysis results to OBS. OBS applies to various scenarios, such as big data analysis, cloud-native application program data, static website hosting, backup/active archive, and deep/cold archive.</p>
|
|
<p id="dli_08_0242__en-us_topic_0125566402_p316664317163">OBS is an object-based storage service. It provides massive, secure, highly reliable, and low-cost data storage capabilities. For more information about OBS, see the <em id="dli_08_0242__dli_08_0236_i13824594127">Object Storage Service Console Operation Guide</em>.</p>
|
|
<div class="note" id="dli_08_0242__note499516812556"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_08_0242__p205169175515">You are advised to use the <a href="dli_08_0267.html">File System Sink Stream (Recommended)</a>.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="dli_08_0242__en-us_topic_0125566402_section426614814471"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_08_0242__en-us_topic_0125566402_p7788427468">Before data exporting, check the version of the OBS bucket. The OBS sink stream supports data exporting to an OBS bucket running OBS 3.0 or a later version.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0242__en-us_topic_0125566402_section12289142974811"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0242__en-us_topic_0125566402_screen1822192119529"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span>
|
|
<span class="normal">16</span>
|
|
<span class="normal">17</span>
|
|
<span class="normal">18</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">stream_id</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"obs"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">field_delimiter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">row_delimiter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">obs_dir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">file_prefix</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">rolling_size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">rolling_interval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">quote</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">array_bracket</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">append</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">max_record_num_per_file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">dump_interval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">dis_notice_channel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span>
|
|
<span class="w"> </span><span class="p">)</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0242__section1435504918486"><h4 class="sectiontitle">Keywords</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0242__en-us_topic_0125566402_table1413253055920" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0242__en-us_topic_0125566402_row121331730155920"><th align="left" class="cellrowborder" valign="top" width="20.52%" id="mcps1.3.4.2.2.4.1.1"><p id="dli_08_0242__en-us_topic_0125566402_p1197243216118">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="8.540000000000001%" id="mcps1.3.4.2.2.4.1.2"><p id="dli_08_0242__en-us_topic_0125566402_p197982468486">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="70.94%" id="mcps1.3.4.2.2.4.1.3"><p id="dli_08_0242__en-us_topic_0125566402_p197215321617">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0242__en-us_topic_0125566402_row131331530185911"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p6133133075914">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p1679844612486">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p9133123016590">Output channel type. <span class="parmvalue" id="dli_08_0242__en-us_topic_0125566402_parmvalue4761221155214"><b>obs</b></span> indicates that data is exported to OBS.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row5133163010597"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p16133183085915">region</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p17989462487">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p11134730115915">Region to which OBS belongs.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__row6947154820405"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__p111537416387">ak</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__p161531941193814">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__p141531741143816">Access Key ID (AK). </p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__row105994620404"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__p58994360384">sk</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__p20899163615388">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__p1989953683812">Secret access key used together with the ID of the access key. </p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row1613423019594"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p119921846105318">encode</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p1479814460485">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p5958175712531">Encoding format. Currently, formats CSV, JSON, ORC, Avro, Avro-Merge, and Parquet are supported.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row19667213145512"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p191346301591">field_delimiter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p1879924612485">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p1013453013593">Separator used to separate every two attributes.</p>
|
|
<p id="dli_08_0242__en-us_topic_0125566402_p1345312290585">This parameter is mandatory only when the CSV encoding format is adopted. If this parameter is not specified, the default separator comma (,) is used.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row1813483017591"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p013443005912">row_delimiter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p0799546134819">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p11341330205912">Row delimiter. This parameter does not need to be configured if the CSV or JSON encoding format is adopted.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row14548212144512"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p870925212283">json_config</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p170985214289">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p12709115262819">If <strong id="dli_08_0242__en-us_topic_0125566402_b9649317137">encode</strong> is set to <strong id="dli_08_0242__en-us_topic_0125566402_b116563117130">json</strong>, you can set this parameter to specify the mapping between the JSON field and the stream definition field. An example of the format is as follows: field1=data_json.field1;field2=data_json.field2.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row171341830155912"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p1297514213553">obs_dir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p179721021185514">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p8971162135516">Directory for storing files. The directory is in the format of {Bucket name}/{Directory name}, for example, obs-a1/dir1/subdir. If <strong id="dli_08_0242__b12102554183918">encode</strong> is set to <strong id="dli_08_0242__b69111446399">csv</strong> (<strong id="dli_08_0242__b214371574120">append</strong> is <strong id="dli_08_0242__b7535104618408">false</strong>), <strong id="dli_08_0242__b268747184014">json</strong> (<strong id="dli_08_0242__b113686207417">append</strong> is <strong id="dli_08_0242__b10288849104014">false</strong>), <strong id="dli_08_0242__b91711215134017">avro_merge</strong>, or <strong id="dli_08_0242__b13479131834015">parquet</strong>, parameterization is supported.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row71341630165914"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p713943185914">file_prefix</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p20138939599">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p51374310592">Prefix of the data export file name. The generated file is named in the format of file_prefix.<em id="dli_08_0242__en-us_topic_0125566402_i1454916569415">x</em>, for example, file_prefix.1 and file_prefix.2. If this parameter is not specified, the file prefix is <strong id="dli_08_0242__en-us_topic_0125566402_b531211322411">temp</strong> by default.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row8144144145918"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p1313573145910">rolling_size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p1813453175916">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p11336345912">Maximum size of a file.</p>
|
|
<div class="note" id="dli_08_0242__en-us_topic_0125566402_note185378521712"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0242__en-us_topic_0125566402_ul8972021426"><li id="dli_08_0242__en-us_topic_0125566402_li1961521627">One or both of <strong id="dli_08_0242__en-us_topic_0125566402_b974310817346">rolling_size</strong> and <strong id="dli_08_0242__en-us_topic_0125566402_b574312843413">rolling_interval</strong> must be configured.</li><li id="dli_08_0242__en-us_topic_0125566402_li179662119210">When the size of a file exceeds the specified size, a new file is generated.</li><li id="dli_08_0242__en-us_topic_0125566402_li1988570192720">The unit can be KB, MB, or GB. If no unit is specified, the byte unit is used.</li><li id="dli_08_0242__en-us_topic_0125566402_li635410473172">This parameter does not need to be configured if the ORC encoding format is adopted.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row59114151504"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p8912415701">rolling_interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p162049362018">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p179122015603">Time mode, in which data is saved to the corresponding directory.</p>
|
|
<div class="note" id="dli_08_0242__en-us_topic_0125566402_note9411172911216"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0242__en-us_topic_0125566402_ul12056144413"><li id="dli_08_0242__en-us_topic_0125566402_li1320518141348">One or both of <strong id="dli_08_0242__en-us_topic_0125566402_b1340716131425">rolling_size</strong> and <strong id="dli_08_0242__en-us_topic_0125566402_b1409141315427">rolling_interval</strong> must be configured.</li><li id="dli_08_0242__en-us_topic_0125566402_li120516141043">After this parameter is specified, data is written to the corresponding directories according to the output time.</li><li id="dli_08_0242__en-us_topic_0125566402_li11205714142">The parameter value can be in the format of yyyy/MM/dd/HH/mm, which is case sensitive. The minimum unit is minute. If this parameter is set to <strong id="dli_08_0242__en-us_topic_0125566402_b13556941104211">yyyy/MM/dd/HH</strong>, data is written to the directory that is generated at the hour time. For example, data generated at 2018-09-10 16:00 will be written to the <strong id="dli_08_0242__en-us_topic_0125566402_b20453841174319">{obs_dir}/2018-09-10_16</strong> directory.</li><li id="dli_08_0242__en-us_topic_0125566402_li720591415418">If both <strong id="dli_08_0242__en-us_topic_0125566402_b65331143124318">rolling_size</strong> and <strong id="dli_08_0242__en-us_topic_0125566402_b2053412437438">rolling_interval</strong> are set, a new file is generated when the size of a single file exceeds the specified size in the directory corresponding to each time point.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row7466184912189"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p846644918182">quote</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p1746654915183">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p246634961814">Modifier, which is added before and after each attribute only when the CSV encoding format is adopted. You are advised to use invisible characters, such as <strong id="dli_08_0242__en-us_topic_0125566402_b17966162915442">u0007</strong>, as the parameter value.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row11851153831917"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p10851838101911">array_bracket</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p7851103814199">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p3851153812191">Array bracket, which can be configured only when the CSV encoding format is adopted. The available options are <strong id="dli_08_0242__en-us_topic_0125566402_b889910249460">()</strong>, <strong id="dli_08_0242__en-us_topic_0125566402_b992691974614">{}</strong>, and <strong id="dli_08_0242__en-us_topic_0125566402_b3300121684619">[]</strong>. For example, if you set this parameter to <strong id="dli_08_0242__en-us_topic_0125566402_b4341737154616">{}</strong>, the array output format is {a1, a2}.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row1440444318819"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p1746311501383">append</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p94637501816">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p1980255719614">The value can be <strong id="dli_08_0242__b150817648991955">true</strong> or <strong id="dli_08_0242__b68974921191955">false</strong>. The default value is <strong id="dli_08_0242__b24805878391955">true</strong>.</p>
|
|
<p id="dli_08_0242__en-us_topic_0125566402_p524216231064">If OBS does not support the append mode and the encoding format is CSV or JSON, set this parameter to <strong id="dli_08_0242__b1090825784018">false</strong>. If <strong id="dli_08_0242__en-us_topic_0125566402_b842352706225124">Append</strong> is set to <strong id="dli_08_0242__en-us_topic_0125566402_b842352706225128">false</strong>, <strong id="dli_08_0242__en-us_topic_0125566402_b842352706225135">max_record_num_per_file</strong> and <strong id="dli_08_0242__en-us_topic_0125566402_b842352706225139">dump_interval</strong> must be set.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row5411122911204"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p241162919202">max_record_num_per_file</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p9411182912015">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p164112296203">Maximum number of records in a file. This parameter needs to be set if <strong id="dli_08_0242__b786415219414">encode</strong> is <strong id="dli_08_0242__b2086918220415">csv</strong> (<strong id="dli_08_0242__b98703294120">append</strong> is <strong id="dli_08_0242__b887017214119">false</strong>), <strong id="dli_08_0242__b1787110214114">json</strong> (<strong id="dli_08_0242__b1787142204117">append</strong> is <strong id="dli_08_0242__b13872921418">false</strong>), <strong id="dli_08_0242__b187212217418">orc</strong>, <strong id="dli_08_0242__b88739214414">avro</strong>, <strong id="dli_08_0242__b187422174115">avro_merge</strong>, or <strong id="dli_08_0242__b1687412184110">parquet</strong>. If the maximum number of records has been reached, a new file is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row41515213217"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p315116220218">dump_interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p81516218218">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p119644815218">Triggering period. This parameter needs to be configured when the ORC encoding format is adopted or notification to DIS is enabled.</p>
|
|
<ul id="dli_08_0242__en-us_topic_0125566402_ul6649101392216"><li id="dli_08_0242__en-us_topic_0125566402_li1764911352220">If the ORC encoding format is specified, this parameter indicates that files will be uploaded to OBS when the triggering period arrives even if the number of file records does not reach the maximum value.</li><li id="dli_08_0242__en-us_topic_0125566402_li1264917137223">In notification to DIS is enabled, this parameter specifies that a notification is sent to DIS every period to indicate that no more files will be generated in the directory.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row1307627162213"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p20307182719229">dis_notice_channel</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p16307127142213">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p18307122712229">DIS channel where DLI sends the record that contains the OBS directory DLI periodically sends the DIS channel a record, which contains the OBS directory, indicating that no more new files will be generated in the directory.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0242__en-us_topic_0125566402_row890411221015"><td class="cellrowborder" valign="top" width="20.52%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0242__en-us_topic_0125566402_p290502216116">encoded_data</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="8.540000000000001%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0242__en-us_topic_0125566402_p139061222015">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="70.94%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0242__en-us_topic_0125566402_p890612214111">Data to be encoded. This parameter is set if <strong id="dli_08_0242__b171211330194112">encode</strong> is <strong id="dli_08_0242__b1812163094119">json</strong> (<strong id="dli_08_0242__b2012233084113">append</strong> is <strong id="dli_08_0242__b1312213094116">false</strong>), <strong id="dli_08_0242__b8122173034113">avro_merge</strong>, or <strong id="dli_08_0242__b191221830174113">parquet</strong>. The format is <strong id="dli_08_0242__b212353064115">${field_name}</strong>, indicating that the stream field content is encoded as a complete record.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0242__section45859307505"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0242__en-us_topic_0125566402_p1648222174615">If a configuration item can be specified through parameter configurations, one or more columns in the record can be used as part of the configuration item. For example, if the configuration item is set to <strong id="dli_08_0242__en-us_topic_0111499977_b842352706185125">car_$ {car_brand}</strong> and the value of <strong id="dli_08_0242__en-us_topic_0111499977_b842352706185132">car_brand</strong> in a record is <strong id="dli_08_0242__en-us_topic_0111499977_b842352706185138">BMW</strong>, the value of this configuration item is <strong id="dli_08_0242__en-us_topic_0111499977_b842352706185147">car_BMW</strong> in the record.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0242__en-us_topic_0125566402_section4673172818499"><h4 class="sectiontitle">Example</h4><ul id="dli_08_0242__en-us_topic_0125566402_ul557820812244"><li id="dli_08_0242__en-us_topic_0125566402_li19578283243">Export the <strong id="dli_08_0242__en-us_topic_0125566402_b10965289290">car_infos</strong> data to the <strong id="dli_08_0242__en-us_topic_0125566402_b99658814295">obs-sink</strong> bucket in OBS. The output directory is <strong id="dli_08_0242__en-us_topic_0125566402_b9965683297">car_infos</strong>. The output file uses <strong id="dli_08_0242__en-us_topic_0125566402_b209650810297">greater_30</strong> as the file name prefix. The maximum size of a single file is 100 MB. If the data size exceeds 100 MB, another new file is generated. The data is encoded in CSV format, the comma (,) is used as the attribute delimiter, and the line break is used as the line separator.<div class="codecoloring" codetype="Sql" id="dli_08_0242__en-us_topic_0125566402_screen26583416316"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span>
|
|
<span class="normal">16</span>
|
|
<span class="normal">17</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">car_infos</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_price</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_timestamp</span><span class="w"> </span><span class="n">LONG</span>
|
|
<span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"obs"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"csv"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"xxx"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">field_delimiter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">","</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">row_delimiter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"\n"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">obs_dir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"obs-sink/car_infos"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">file_prefix</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"greater_30"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">rolling_size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"100m"</span>
|
|
<span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li></ul>
|
|
<ul id="dli_08_0242__en-us_topic_0125566402_ul9213191312411"><li id="dli_08_0242__en-us_topic_0125566402_li721391316248">Example of the ORC encoding format<div class="codecoloring" codetype="Sql" id="dli_08_0242__en-us_topic_0125566402_screen16758115920232"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span>
|
|
<span class="normal">16</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">car_infos</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_brand</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_price</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_timestamp</span><span class="w"> </span><span class="n">LONG</span>
|
|
<span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"obs"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"xxx"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"orc"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">obs_dir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"dli-append-2/obsorc"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">FILE_PREFIX</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"es_info"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">max_record_num_per_file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"100000"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">dump_interval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"60"</span>
|
|
<span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</li><li id="dli_08_0242__li1461221210492">For details about the parquet encoding example, see the example in <a href="dli_08_0267.html">File System Sink Stream (Recommended)</a>.</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0240.html">Creating a Sink Stream</a></div>
|
|
</div>
|
|
</div>
|
|
|