forked from docs/doc-exports
Su, Xiaomeng
76a5b1ee83
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
148 lines
18 KiB
HTML
148 lines
18 KiB
HTML
<a name="dli_08_0243"></a><a name="dli_08_0243"></a>
|
|
|
|
<h1 class="topictitle1">CloudTable HBase Sink Stream</h1>
|
|
<div id="body1574393083137"><div class="section" id="dli_08_0243__en-us_topic_0111555124_section12877153413613"><h4 class="sectiontitle">Function</h4><p id="dli_08_0243__en-us_topic_0111555124_p10804710162011">DLI exports the job output data to HBase of CloudTable. HBase is a column-oriented distributed cloud storage system that features enhanced reliability, excellent performance, and elastic scalability. It applies to the storage of massive amounts of data and distributed computing. You can use HBase to build a storage system capable of storing TB- or even PB-level data. With HBase, you can filter and analyze data with ease and get responses in milliseconds, rapidly mining data value. Structured and semi-structured key-value data can be stored, including messages, reports, recommendation data, risk control data, logs, and orders. With DLI, you can write massive volumes of data to HBase at a high speed and with low latency.</p>
|
|
<p id="dli_08_0243__en-us_topic_0111555124_p1932235815353">CloudTable is a distributed, scalable, and fully-hosted key-value data storage service based on Apache HBase. It provides DLI with high-performance random read and write capabilities, which are helpful when applications need to store and query a massive amount of structured data, semi-structured data, and time series data. CloudTable applies to IoT scenarios and storage and query of massive volumes of key-value data. For more information about CloudTable, see the <em id="dli_08_0243__i1338164413139">CloudTable Service User Guide</em>.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0243__section1144171011544"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_08_0243__p1038754112542">In this scenario, jobs must run on the dedicated queue of DLI. Therefore, DLI must interconnect with the enhanced datasource connection that has been connected with CloudTable HBase. You can also set the security group rules as required.</p>
|
|
<p id="dli_08_0243__p94091618103916">For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_08_0243__dli_08_0249_en-us_topic_0132966325_filepath13726182511815"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_08_0243__dli_08_0249_cite534516e564224b13bad1de4471da3139181527">Data Lake Insight User Guide</cite></i>.</p>
|
|
<p id="dli_08_0243__p10832135481415">For details about how to configure security group rules, see <span class="filepath" id="dli_08_0243__dli_08_0249_filepath78235157113"><b>Security Group</b></span> in the <em id="dli_08_0243__dli_08_0249_i082519153111">Virtual Private Cloud User Guide</em>.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0243__en-us_topic_0111555124_section15771218124216"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0243__screen746013311958"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
|
|
<span class="normal">2</span>
|
|
<span class="normal">3</span>
|
|
<span class="normal">4</span>
|
|
<span class="normal">5</span>
|
|
<span class="normal">6</span>
|
|
<span class="normal">7</span>
|
|
<span class="normal">8</span>
|
|
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">stream_id</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"cloudtable"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">cluster_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">table_columns</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">create_if_not_exist</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span>
|
|
<span class="w"> </span><span class="p">)</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0243__section412418440612"><h4 class="sectiontitle">Keywords</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0243__en-us_topic_0111555124_table754518122117" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0243__en-us_topic_0111555124_row1455917811212"><th align="left" class="cellrowborder" valign="top" width="18.75%" id="mcps1.3.4.2.2.4.1.1"><p id="dli_08_0243__en-us_topic_0111555124_p5563482212">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="9.46%" id="mcps1.3.4.2.2.4.1.2"><p id="dli_08_0243__en-us_topic_0111555124_p1957018822115">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="71.78999999999999%" id="mcps1.3.4.2.2.4.1.3"><p id="dli_08_0243__en-us_topic_0111555124_p185743812119">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0243__en-us_topic_0111555124_row157918812211"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0243__en-us_topic_0111555124_p05834892115">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0243__en-us_topic_0111555124_p4587382218">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.78999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0243__en-us_topic_0111555124_p19591198182118">Output channel type. <span class="parmvalue" id="dli_08_0243__en-us_topic_0111555124_parmvalue182321329537"><b>cloudtable</b></span> indicates that data is exported to CloudTable (HBase).</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0243__en-us_topic_0111555124_row359578112117"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0243__en-us_topic_0111555124_p25991982217">region</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0243__en-us_topic_0111555124_p9602280211">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.78999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0243__en-us_topic_0111555124_p1960788142117">Region to which CloudTable belongs.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0243__en-us_topic_0111555124_row36092814214"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0243__en-us_topic_0111555124_p1161410882112">cluster_id</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0243__en-us_topic_0111555124_p561716842119">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.78999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0243__en-us_topic_0111555124_p16621980218">ID of the cluster to which the data you want to insert belongs.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0243__en-us_topic_0111555124_row106231380212"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0243__en-us_topic_0111555124_p1362608122111">table_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0243__en-us_topic_0111555124_p16632178162114">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.78999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0243__en-us_topic_0111555124_p1063778162112">Name of the table, into which data is to be inserted. It can be specified through parameter configurations. For example, if you want one or more certain columns as part of the table name, use <strong id="dli_08_0243__en-us_topic_0111555124_b842352706114129">car_pass_inspect_with_age_${car_age}</strong>, where <strong id="dli_08_0243__en-us_topic_0111555124_b842352706114136">car_age</strong> is the column name.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0243__en-us_topic_0111555124_row136395814212"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0243__en-us_topic_0111555124_p10645198152118">table_columns</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0243__en-us_topic_0111555124_p66492872113">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.78999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0243__en-us_topic_0111555124_p10653168142117">Columns to be inserted. The format is <strong id="dli_08_0243__en-us_topic_0111555124_b842352706184830">rowKey, f1:c1, f1:c2, f2:c1</strong>, where <strong id="dli_08_0243__en-us_topic_0111555124_b842352706184832">rowKey</strong> must be specified. If you do not want to add a column (for example, the third column) to the database, set this parameter to <strong id="dli_08_0243__en-us_topic_0111555124_b842352706184823">rowKey,f1:c1,,f2:c1</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0243__en-us_topic_0111555124_row18255112224616"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0243__en-us_topic_0111555124_p1425612234615">illegal_data_table</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0243__en-us_topic_0111555124_p825652213463">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.78999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0243__en-us_topic_0111555124_p3174143774613">If this parameter is specified, abnormal data (for example, <strong id="dli_08_0243__en-us_topic_0111555124_b842352706184919">rowKey</strong> does not exist) will be written into the table. If not specified, abnormal data will be discarded. The rowKey value is a timestamp followed by six random digits, and the schema is info:data, info:reason.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0243__en-us_topic_0111555124_row186552862111"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0243__en-us_topic_0111555124_p2812125259">create_if_not_exist</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0243__en-us_topic_0111555124_p1766298192112">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.78999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0243__en-us_topic_0111555124_p4974422202518">Whether to create a table or column into which the data is written when this table or column does not exist. The value can be <strong id="dli_08_0243__en-us_topic_0111555124_b84235270693447">true</strong> or <strong id="dli_08_0243__en-us_topic_0111555124_b84235270693450">false</strong>. The default value is <strong id="dli_08_0243__b13445184017447">false</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0243__en-us_topic_0111555124_row645849194613"><td class="cellrowborder" valign="top" width="18.75%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0243__en-us_topic_0111555124_p164724914610">batch_insert_data_num</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0243__en-us_topic_0111555124_p1247849164617">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.78999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0243__en-us_topic_0111555124_p44724944616">Number of data records to be written in batches at a time. The value must be a positive integer. The upper limit is <strong id="dli_08_0243__en-us_topic_0111555124_b84235270619346">100</strong>. The default value is <strong id="dli_08_0243__en-us_topic_0111555124_b842352706193359">10</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0243__section1942141010711"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0243__en-us_topic_0111555124_ul53424216402"><li id="dli_08_0243__en-us_topic_0111555124_li18342122113401">If a configuration item can be specified through parameter configurations, one or more columns in the record can be used as part of the configuration item. For example, if the configuration item is set to <strong id="dli_08_0243__b185565283493">car_$ {car_brand}</strong> and the value of <strong id="dli_08_0243__b6557202818490">car_brand</strong> in a record is <strong id="dli_08_0243__b655718286498">BMW</strong>, the value of this configuration item is <strong id="dli_08_0243__b8557928134917">car_BMW</strong> in the record.</li><li id="dli_08_0243__li179015117279">In this way, data is written to HBase of CloudTable. The speed is limited. The dedicated resource mode is recommended.</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0243__en-us_topic_0111555124_section34021023184217"><h4 class="sectiontitle">Example</h4><p id="dli_08_0243__en-us_topic_0111555124_p127151381217">Output data of stream <strong id="dli_08_0243__en-us_topic_0111555124_b84235270693644">qualified_cars</strong> to CloudTable (HBase).</p>
|
|
<div class="codecoloring" codetype="Sql" id="dli_08_0243__en-us_topic_0111555124_screen4717587212"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span>
|
|
<span class="normal">16</span>
|
|
<span class="normal">17</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">qualified_cars</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_age</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">average_speed</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">total_miles</span><span class="w"> </span><span class="nb">INT</span>
|
|
<span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"cloudtable"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"xxx"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">cluster_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"209ab1b6-de25-4c48-8e1e-29e09d02de28"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"car_pass_inspect_with_age_${car_age}"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">table_columns</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"rowKey,info:owner,,car:speed,car:miles"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">illegal_data_table</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"illegal_data"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">create_if_not_exist</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"true"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">batch_insert_data_num</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"20"</span>
|
|
<span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0240.html">Creating a Sink Stream</a></div>
|
|
</div>
|
|
</div>
|
|
|