forked from docs/doc-exports
Su, Xiaomeng
76a5b1ee83
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
164 lines
18 KiB
HTML
164 lines
18 KiB
HTML
<a name="dli_08_0255"></a><a name="dli_08_0255"></a>
|
|
|
|
<h1 class="topictitle1">MRS HBase Sink Stream</h1>
|
|
<div id="body1574393117323"><div class="section" id="dli_08_0255__section1114513223324"><h4 class="sectiontitle">Function</h4><p id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_p17940172645314">DLI exports the output data of the Flink job to HBase of MRS.</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0255__section88881339193311"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_ul19567726133916"><li id="dli_08_0255__en-us_topic_0154786456_li18306341117">An MRS cluster has been created by using your account. DLI can interconnect with HBase clusters with Kerberos enabled.</li><li id="dli_08_0255__en-us_topic_0154786456_li1380511910163">In this scenario, jobs must run on the dedicated queue of DLI. Ensure that the dedicated queue of DLI has been created.</li><li id="dli_08_0255__en-us_topic_0154786456_li19621034131214">Ensure that a datasource connection has been set up between the DLI dedicated queue and the MRS cluster, and security group rules have been configured based on the site requirements.<p id="dli_08_0255__p94091618103916"><a name="dli_08_0255__en-us_topic_0154786456_li19621034131214"></a><a name="en-us_topic_0154786456_li19621034131214"></a>For details about how to create an enhanced datasource connection, see <span class="filepath" id="dli_08_0255__dli_08_0249_en-us_topic_0132966325_filepath13726182511815"><b>Enhanced Datasource Connections</b></span> in the <i><cite id="dli_08_0255__dli_08_0249_cite534516e564224b13bad1de4471da3139181527">Data Lake Insight User Guide</cite></i>.</p>
|
|
<p id="dli_08_0255__p17953337436">For details about how to configure security group rules, see <span class="filepath" id="dli_08_0255__dli_08_0249_filepath78235157113"><b>Security Group</b></span> in the <em id="dli_08_0255__dli_08_0249_i082519153111">Virtual Private Cloud User Guide</em>.</p>
|
|
</li><li id="dli_08_0255__en-us_topic_0154786456_li141601247124415"><strong id="dli_08_0255__b18945184732717">If you use MRS HBase, ensure that you have added IP addresses of all hosts in the MRS cluster for the enhanced datasource connection.</strong><p id="dli_08_0255__en-us_topic_0154786456_p12685121631310">For details about how to add an IP-domain mapping, see <span class="filepath" id="dli_08_0255__filepath164721096611"><b>Datasource Connections >Enhanced Datasource Connections > Modifying the Host Information</b></span> in the <i><cite id="dli_08_0255__citede9f5e80eb494fd08e33e7a71a4f8bc9181530">Data Lake Insight User Guide</cite></i>.</p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0255__section874131063412"><h4 class="sectiontitle">Syntax</h4><div class="codecoloring" codetype="Sql" id="dli_08_0255__screen686194914313"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">stream_id</span><span class="w"> </span><span class="p">(</span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="w"> </span><span class="p">(</span><span class="s1">','</span><span class="w"> </span><span class="n">attr_name</span><span class="w"> </span><span class="n">attr_type</span><span class="p">)</span><span class="o">*</span><span class="w"> </span><span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"mrs_hbase"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">cluster_address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">table_columns</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">illegal_data_table</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">batch_insert_data_num</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">action</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">""</span>
|
|
<span class="p">)</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0255__section955162414520"><h4 class="sectiontitle">Keywords</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_table2972926145319" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_row29913265537"><th align="left" class="cellrowborder" valign="top" width="18.39%" id="mcps1.3.4.2.2.4.1.1"><p id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_p17998132611534">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="16.46%" id="mcps1.3.4.2.2.4.1.2"><p id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_p12310279537">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="65.14999999999999%" id="mcps1.3.4.2.2.4.1.3"><p id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_p10982717536">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_row816152795316"><td class="cellrowborder" valign="top" width="18.39%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_p202362713532">type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_p6281227165310">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.14999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_p143216277530">Output channel type. <strong id="dli_08_0255__b84235270692937">mrs_hbase</strong> indicates that data is exported to HBase of MRS.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0255__en-us_topic_0154786456_row11622840193411"><td class="cellrowborder" valign="top" width="18.39%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0255__en-us_topic_0154786456_p12623144016340">region</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0255__en-us_topic_0154786456_p862384013340">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.14999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0255__en-us_topic_0154786456_p206108919489">Region where MRS resides.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0255__en-us_topic_0154786456_row17629134623411"><td class="cellrowborder" valign="top" width="18.39%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0255__en-us_topic_0154786456_p862914612343">cluster_address</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0255__en-us_topic_0154786456_p96292465342">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.14999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0255__en-us_topic_0154786456_p1562964663414">ZooKeeper address of the cluster to which the data table to be inserted belongs. The format is <strong id="dli_08_0255__b0447141705">ip1,ip2:port</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0255__en-us_topic_0154786456_row10886849163412"><td class="cellrowborder" valign="top" width="18.39%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0255__en-us_topic_0154786456_p16886104914341">table_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0255__en-us_topic_0154786456_p2886154915347">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.14999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0255__en-us_topic_0154786456_p1241183712375">Name of the table where data is to be inserted.</p>
|
|
<p id="dli_08_0255__en-us_topic_0154786456_p48868492345">It can be specified through parameter configurations. For example, if you want one or more certain columns as part of the table name, use <strong id="dli_08_0255__b842352706114129">car_pass_inspect_with_age_${car_age}</strong>, where <strong id="dli_08_0255__b842352706114136">car_age</strong> is the column name.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_row0343279539"><td class="cellrowborder" valign="top" width="18.39%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0255__en-us_topic_0154786456_p67212556345">table_columns</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0255__en-us_topic_0154786456_p1772955163419">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.14999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0255__en-us_topic_0154786456_p370185517345">Columns to be inserted. The format is <strong id="dli_08_0255__b2163985016">rowKey, f1:c1, f1:c2, f2:c1</strong>, where <strong id="dli_08_0255__b14169168200">rowKey</strong> must be specified. If you do not want to add a column (for example, the third column) to the database, set this parameter to <strong id="dli_08_0255__b171701782012">rowKey,f1:c1,,f2:c1</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_row955727195316"><td class="cellrowborder" valign="top" width="18.39%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0255__en-us_topic_0154786456_p13671055143410">illegal_data_table</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0255__en-us_topic_0154786456_p196515510344">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.14999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0255__en-us_topic_0154786456_p06211551343">If this parameter is specified, abnormal data (for example, <strong id="dli_08_0255__b842352706184919">rowKey</strong> does not exist) will be written into the table. If not specified, abnormal data will be discarded. The <strong id="dli_08_0255__b18994155831318">rowKey</strong> value is <strong id="dli_08_0255__b11974143101416">taskNo_</strong><em id="dli_08_0255__i61904618145">Timestamp</em> followed by six random digits, and the schema is info:data, info:reason.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_row871152765314"><td class="cellrowborder" valign="top" width="18.39%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0255__en-us_topic_0154786456_p86195533415">batch_insert_data_num</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0255__en-us_topic_0154786456_p15935512341">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.14999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0255__en-us_topic_0154786456_p357195517342">Number of data records to be written in batches at a time. The value must be a positive integer. The upper limit is <strong id="dli_08_0255__b1980992014357">1000</strong>. The default value is <strong id="dli_08_0255__b1781092017358">10</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0255__en-us_topic_0154786456_row5326317143517"><td class="cellrowborder" valign="top" width="18.39%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0255__en-us_topic_0154786456_p23262176357">action</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0255__en-us_topic_0154786456_p1432631723516">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.14999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0255__en-us_topic_0154786456_p1232621763511">Whether data is added or deleted. Available options include <strong id="dli_08_0255__b119231623184018">add</strong> and <strong id="dli_08_0255__b89266259405">delete</strong>. The default value is <strong id="dli_08_0255__b166511028164014">add</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0255__row2912122613012"><td class="cellrowborder" valign="top" width="18.39%" headers="mcps1.3.4.2.2.4.1.1 "><p id="dli_08_0255__p1814431414555">krb_auth</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.46%" headers="mcps1.3.4.2.2.4.1.2 "><p id="dli_08_0255__p131442148552">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.14999999999999%" headers="mcps1.3.4.2.2.4.1.3 "><p id="dli_08_0255__p1942105974614">Authentication name for creating a datasource connection authentication. This parameter is mandatory when Kerberos authentication is enabled. Set this parameter to the corresponding cross-source authentication name. </p>
|
|
<div class="note" id="dli_08_0255__note1113202611465"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="dli_08_0255__p19135265469">Ensure that the <span class="filepath" id="dli_08_0255__filepath138881940904"><b>/etc/hosts</b></span> information of the master node in the MRS cluster is added to the host file of the DLI queue.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0255__section926920566455"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0255__p138663217461">None</p>
|
|
</div>
|
|
<div class="section" id="dli_08_0255__section517474233416"><h4 class="sectiontitle">Example</h4><p id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_p4119112712532">Output data to HBase of MRS.</p>
|
|
<div class="codecoloring" codetype="Sql" id="dli_08_0255__en-us_topic_0154786456_en-us_topic_0111562554_screen9122162775318"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
|
|
<span class="normal"> 2</span>
|
|
<span class="normal"> 3</span>
|
|
<span class="normal"> 4</span>
|
|
<span class="normal"> 5</span>
|
|
<span class="normal"> 6</span>
|
|
<span class="normal"> 7</span>
|
|
<span class="normal"> 8</span>
|
|
<span class="normal"> 9</span>
|
|
<span class="normal">10</span>
|
|
<span class="normal">11</span>
|
|
<span class="normal">12</span>
|
|
<span class="normal">13</span>
|
|
<span class="normal">14</span>
|
|
<span class="normal">15</span>
|
|
<span class="normal">16</span>
|
|
<span class="normal">17</span>
|
|
<span class="normal">18</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="n">SINK</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">qualified_cars</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">car_age</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">average_speed</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">total_miles</span><span class="w"> </span><span class="nb">INT</span>
|
|
<span class="p">)</span>
|
|
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
|
|
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"mrs_hbase"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"xxx"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">cluster_address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"192.16.0.88,192.87.3.88:2181"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="k">table_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"car_pass_inspect_with_age_${car_age}"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">table_columns</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"rowKey,info:owner,,car:speed,car:miles"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">illegal_data_table</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"illegal_data"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">batch_insert_data_num</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"20"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">action</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"add"</span><span class="p">,</span>
|
|
<span class="w"> </span><span class="n">krb_auth</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">"KRB_AUTH_NAME"</span>
|
|
<span class="w"> </span><span class="p">);</span>
|
|
</pre></div></td></tr></table></div>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0240.html">Creating a Sink Stream</a></div>
|
|
</div>
|
|
</div>
|
|
|