vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0396.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

361 lines
37 KiB
HTML
Raw Blame History

<a name="dli_08_0396"></a><a name="dli_08_0396"></a>
<h1 class="topictitle1">HBase Result Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0396__en-us_topic_0000001310215789_section552112010617"><h4 class="sectiontitle">Function</h4><p id="dli_08_0396__en-us_topic_0000001310215789_p48281851565">DLI outputs the job data to HBase. HBase is a column-oriented distributed cloud storage system that features enhanced reliability, excellent performance, and elastic scalability. It applies to the storage of massive amounts of data and distributed computing. You can use HBase to build a storage system capable of storing TB- or even PB-level data. With HBase, you can filter and analyze data with ease and get responses in milliseconds, rapidly mining data value. Structured and semi-structured key-value data can be stored, including messages, reports, recommendation data, risk control data, logs, and orders. With DLI, you can write massive volumes of data to HBase at a high speed and with low latency.</p>
</div>
<div class="section" id="dli_08_0396__en-us_topic_0000001310215789_section101912311388"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0396__en-us_topic_0000001310215789_ul96952193162"><li id="dli_08_0396__en-us_topic_0000001310215789_li76957193160">An enhanced datasource connection has been created for DLI to connect to HBase, so that jobs can run on the dedicated queue of DLI and you can set the security group rules as required.
</li><li id="dli_08_0396__en-us_topic_0000001310215789_li168891128102320">If MRS HBase is used, IP addresses of all hosts in the MRS cluster have been added to host information of the enhanced datasource connection.<p id="dli_08_0396__en-us_topic_0000001310215789_p1435791362116"><a name="dli_08_0396__en-us_topic_0000001310215789_li168891128102320"></a><a name="en-us_topic_0000001310215789_li168891128102320"></a>.</p>
</li></ul>
</div>
<div class="section" id="dli_08_0396__en-us_topic_0000001310215789_section12242649121615"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0396__en-us_topic_0000001310215789_ul322397111719"><li id="dli_08_0396__en-us_topic_0000001310215789_li14781724112416">When creating a Flink OpenSource SQL job, you need to set <strong id="dli_08_0396__en-us_topic_0000001310215789_b1832381703615">Flink Version</strong> to <strong id="dli_08_0396__en-us_topic_0000001310215789_b6324101716365">1.12</strong> on the <strong id="dli_08_0396__en-us_topic_0000001310215789_b1532481714368">Running Parameters</strong> tab of the job editing page, select <strong id="dli_08_0396__en-us_topic_0000001310215789_b13325191743613">Save Job Log</strong>, and set the OBS bucket for saving job logs.</li><li id="dli_08_0396__en-us_topic_0000001310215789_li1896586122310">The column families in created HBase result table must be declared as the ROW type, the field names map the column family names, and the nested field names map the column qualifier names. There is no need to declare all the families and qualifiers in the schema. Users can declare what is used in the query. Except the ROW type fields, the single atomic type field (for example, STRING or BIGINT) will be recognized as the HBase rowkey. The rowkey field can be an arbitrary name, but should be quoted using backticks if it is a reserved keyword.</li></ul>
</div>
<div class="section" id="dli_08_0396__en-us_topic_0000001310215789_section1953417165276"><h4 class="sectiontitle">Syntax</h4><pre class="screen" id="dli_08_0396__en-us_topic_0000001310215789_screen1663134719274">create table hbaseSink (
attr_name attr_type
(',' attr_name attr_type)*
','PRIMARY KEY (attr_name, ...) NOT ENFORCED)
) with (
'connector' = 'hbase-2.2',
'table-name' = '',
'zookeeper.quorum' = ''
);</pre>
</div>
<div class="section" id="dli_08_0396__en-us_topic_0000001310215789_section590310474382"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0396__en-us_topic_0000001310215789_table10428102616404" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0396__en-us_topic_0000001310215789_row14295269403"><th align="left" class="cellrowborder" valign="top" width="16.55165516551655%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0396__en-us_topic_0000001310215789_p1642914263402">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="7.01070107010701%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0396__en-us_topic_0000001310215789_p1642992611405">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="11.37113711371137%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0396__en-us_topic_0000001310215789_p32071110142315">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10.491049104910491%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0396__en-us_topic_0000001310215789_p143311133232">Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="54.57545754575457%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0396__en-us_topic_0000001310215789_p19429182617401">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0396__en-us_topic_0000001310215789_row1242914267404"><td class="cellrowborder" valign="top" width="16.55165516551655%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p393065810586">connector</p>
</td>
<td class="cellrowborder" valign="top" width="7.01070107010701%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p12930175813580">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.37113711371137%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1511511373714">None</p>
</td>
<td class="cellrowborder" valign="top" width="10.491049104910491%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0396__en-us_topic_0000001310215789_p129491421378">String</p>
</td>
<td class="cellrowborder" valign="top" width="54.57545754575457%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0396__en-us_topic_0000001310215789_p181621120191118">Connector to be used. Set this parameter to <strong id="dli_08_0396__en-us_topic_0000001310215789_b235010281400">hbase-2.2</strong>.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row1942982614405"><td class="cellrowborder" valign="top" width="16.55165516551655%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p199301358135815">table-name</p>
</td>
<td class="cellrowborder" valign="top" width="7.01070107010701%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p193085835811">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.37113711371137%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0396__en-us_topic_0000001310215789_p101151637974">None</p>
</td>
<td class="cellrowborder" valign="top" width="10.491049104910491%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0396__en-us_topic_0000001310215789_p79495424713">String</p>
</td>
<td class="cellrowborder" valign="top" width="54.57545754575457%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0396__en-us_topic_0000001310215789_p119302058165815">Name of the HBase table to connect.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row154291526134017"><td class="cellrowborder" valign="top" width="16.55165516551655%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p093015817586">zookeeper.quorum</p>
</td>
<td class="cellrowborder" valign="top" width="7.01070107010701%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p89301358155814">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="11.37113711371137%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0396__en-us_topic_0000001310215789_p101157370719">None</p>
</td>
<td class="cellrowborder" valign="top" width="10.491049104910491%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0396__en-us_topic_0000001310215789_p17949742671">String</p>
</td>
<td class="cellrowborder" valign="top" width="54.57545754575457%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0396__en-us_topic_0000001310215789_p9871653181210">HBase ZooKeeper instance information, in the format of ZookeeperAddress:ZookeeperPort.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p20696122916251">The following uses an MRS HBase cluster as an example to describe how to obtain the IP address and port number of ZooKeeper used by this parameter:</p>
<ul id="dli_08_0396__en-us_topic_0000001310215789_ul571954114435"><li id="dli_08_0396__en-us_topic_0000001310215789_li177201341154318">On MRS Manager, choose <strong id="dli_08_0396__en-us_topic_0000001310215789_b1384111210428">Cluster</strong> and click the name of the desired cluster. Choose <strong id="dli_08_0396__en-us_topic_0000001310215789_b684111214216">Services</strong> &gt; <strong id="dli_08_0396__en-us_topic_0000001310215789_b17842161224218">ZooKeeper</strong> &gt; <strong id="dli_08_0396__en-us_topic_0000001310215789_b13843201244210">Instance</strong>, and obtain the IP address of the ZooKeeper instance.</li><li id="dli_08_0396__en-us_topic_0000001310215789_li1433443194310">On MRS Manager, choose <strong id="dli_08_0396__en-us_topic_0000001310215789_b12939325144218">Cluster</strong> and click the name of the desired cluster. Choose <strong id="dli_08_0396__en-us_topic_0000001310215789_b16940325124218">Services</strong> &gt; <strong id="dli_08_0396__en-us_topic_0000001310215789_b1394112574217">ZooKeeper</strong> &gt; <strong id="dli_08_0396__en-us_topic_0000001310215789_b1794119257427">Configurations</strong> &gt; <strong id="dli_08_0396__en-us_topic_0000001310215789_b8942125124214">All Configurations</strong>, search for the <strong id="dli_08_0396__en-us_topic_0000001310215789_b3943122515426">clientPort</strong> parameter, and obtain its value, that is, the ZooKeeper port number.</li></ul>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row04293262407"><td class="cellrowborder" valign="top" width="16.55165516551655%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1193025875815">zookeeper.znode.parent</p>
</td>
<td class="cellrowborder" valign="top" width="7.01070107010701%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p3930205865811">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.37113711371137%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1911618371278">/hbase</p>
</td>
<td class="cellrowborder" valign="top" width="10.491049104910491%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1694912422710">String</p>
</td>
<td class="cellrowborder" valign="top" width="54.57545754575457%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0396__en-us_topic_0000001310215789_p17930195815586">Root directory in ZooKeeper. The default value is <strong id="dli_08_0396__en-us_topic_0000001310215789_b13581104016423">/hbase</strong>.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row542942674010"><td class="cellrowborder" valign="top" width="16.55165516551655%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p898164614212">null-string-literal</p>
</td>
<td class="cellrowborder" valign="top" width="7.01070107010701%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p39874618217">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.37113711371137%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0396__en-us_topic_0000001310215789_p298174618218">null</p>
</td>
<td class="cellrowborder" valign="top" width="10.491049104910491%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0396__en-us_topic_0000001310215789_p998646182118">String</p>
</td>
<td class="cellrowborder" valign="top" width="54.57545754575457%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0396__en-us_topic_0000001310215789_p3957115093311">Representation for null values for string fields.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p1981246142115">The HBase sink encodes/decodes empty bytes as null values for all types except the string type.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row84294262402"><td class="cellrowborder" valign="top" width="16.55165516551655%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1582920410254">sink.buffer-flush.max-size</p>
</td>
<td class="cellrowborder" valign="top" width="7.01070107010701%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p18829740257">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.37113711371137%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1482910492517">2mb</p>
</td>
<td class="cellrowborder" valign="top" width="10.491049104910491%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0396__en-us_topic_0000001310215789_p282915452515">MemorySize</p>
</td>
<td class="cellrowborder" valign="top" width="54.57545754575457%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0396__en-us_topic_0000001310215789_p7392829343">Maximum size in memory of buffered rows for each write request.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p1287516310341">This can improve performance for writing data to the HBase database, but may increase the latency.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p98295414253">You can set this parameter to <strong id="dli_08_0396__en-us_topic_0000001310215789_b1472519564445">0</strong> to disable it.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row16429112624010"><td class="cellrowborder" valign="top" width="16.55165516551655%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1714417156254">sink.buffer-flush.max-rows</p>
</td>
<td class="cellrowborder" valign="top" width="7.01070107010701%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p61445152252">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.37113711371137%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1914461562517">1000</p>
</td>
<td class="cellrowborder" valign="top" width="10.491049104910491%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0396__en-us_topic_0000001310215789_p17144131542510">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="54.57545754575457%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0396__en-us_topic_0000001310215789_p694418633417">Maximum number of rows to buffer for each write request.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p710279123415">This can improve performance for writing data to the HBase database, but may increase the latency.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p814431513251">You can set this parameter to <strong id="dli_08_0396__en-us_topic_0000001310215789_b34436399453">0</strong> to disable it.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row34292268400"><td class="cellrowborder" valign="top" width="16.55165516551655%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p15618103610259">sink.buffer-flush.interval</p>
</td>
<td class="cellrowborder" valign="top" width="7.01070107010701%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p19943163782610">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.37113711371137%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0396__en-us_topic_0000001310215789_p146181736162513">1s</p>
</td>
<td class="cellrowborder" valign="top" width="10.491049104910491%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0396__en-us_topic_0000001310215789_p861814369254">Duration</p>
</td>
<td class="cellrowborder" valign="top" width="54.57545754575457%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0396__en-us_topic_0000001310215789_p471111393417">Interval for refreshing the buffer, during which data is refreshed by asynchronous threads.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p1653314155342">This can improve performance for writing data to the HBase database, but may increase the latency.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p79911517173418">You can set this parameter to <strong id="dli_08_0396__en-us_topic_0000001310215789_b467822044612">0</strong> to disable it.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p7618436192517">Note: If <strong id="dli_08_0396__en-us_topic_0000001310215789_b74056133514">sink.buffer-flush.max-size</strong> and <strong id="dli_08_0396__en-us_topic_0000001310215789_b104016163516">sink.buffer-flush.max-rows</strong> are both set to <strong id="dli_08_0396__en-us_topic_0000001310215789_b11417613511">0</strong> and the buffer refresh interval is configured, the buffer is asynchronously refreshed.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p11649135219406">The format is <em id="dli_08_0396__en-us_topic_0000001310215789_i1577641103620">{length value}{time unit label}</em>, for example, <strong id="dli_08_0396__en-us_topic_0000001310215789_b7217148153519">123ms, 321s</strong>. The supported time units include <strong id="dli_08_0396__en-us_topic_0000001310215789_b12845191033614">d</strong>, <strong id="dli_08_0396__en-us_topic_0000001310215789_b12543313173618">h</strong>, <strong id="dli_08_0396__en-us_topic_0000001310215789_b02501157365">min</strong>, <strong id="dli_08_0396__en-us_topic_0000001310215789_b20184817143614">s</strong>, and <strong id="dli_08_0396__en-us_topic_0000001310215789_b4389619103620">ms</strong> (default unit).</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row114291526124015"><td class="cellrowborder" valign="top" width="16.55165516551655%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p11314154782518">sink.parallelism</p>
</td>
<td class="cellrowborder" valign="top" width="7.01070107010701%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p831418477258">No</p>
</td>
<td class="cellrowborder" valign="top" width="11.37113711371137%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0396__en-us_topic_0000001310215789_p43141547102516">None</p>
</td>
<td class="cellrowborder" valign="top" width="10.491049104910491%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0396__en-us_topic_0000001310215789_p5314114715257">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="54.57545754575457%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1395213843411">Defines the parallelism of the HBase sink operator.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p0314154710257">By default, the parallelism is determined by the framework using the same parallelism of the upstream chained operator.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0396__en-us_topic_0000001310215789_section11151124192013"><h4 class="sectiontitle">Data Type Mapping</h4><p id="dli_08_0396__en-us_topic_0000001310215789_p169933132115">HBase stores all data as byte arrays. The data needs to be serialized and deserialized during read and write operations.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p66992342118">When serializing and de-serializing, Flink HBase connector uses utility class <strong id="dli_08_0396__en-us_topic_0000001310215789_b1895134624811">org.apache.hadoop.hbase.util.Bytes</strong> provided by HBase (Hadoop) to convert Flink data types to and from byte arrays.</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p269919319215">Flink HBase connector encodes null values to empty bytes, and decode empty bytes to null values for all data types except the string type. For the string type, the null literal is determined by the <strong id="dli_08_0396__en-us_topic_0000001310215789_b1551072674916">null-string-literal</strong> option.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0396__en-us_topic_0000001310215789_table17452636132111" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Data type mapping</caption><thead align="left"><tr id="dli_08_0396__en-us_topic_0000001310215789_row1345263617217"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.6.5.2.3.1.1"><p id="dli_08_0396__en-us_topic_0000001310215789_p16452436132117">Flink SQL Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.6.5.2.3.1.2"><p id="dli_08_0396__en-us_topic_0000001310215789_p12452123662114">HBase Conversion</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0396__en-us_topic_0000001310215789_row14452636132115"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p2045273672116">CHAR / VARCHAR / STRING</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p427912322519">byte[] toBytes(String s)</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p12279192302515">String toString(byte[] b)</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row1145216365211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p19452143632119">BOOLEAN</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p759018125250">byte[] toBytes(boolean b)</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p4590212192515">boolean toBoolean(byte[] b)</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row17452153612211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p18452103611212">BINARY / VARBINARY</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p14452736182115">Returns byte[] as is.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row34521365219"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p13452173612116">DECIMAL</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1214719587243">byte[] toBytes(BigDecimal v)</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p614745818244">BigDecimal toBigDecimal(byte[] b)</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row194521636162114"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p94521636202112">TINYINT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1211974622411">new byte[] { val }</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p1119174610240">bytes[0] // returns first and only byte from bytes</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row1452183613212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1452133616212">SMALLINT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1454183510245">byte[] toBytes(short val)</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p17541435142410">short toShort(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row445243617216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p2452636112110">INT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1320111234249">byte[] toBytes(int val)</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p18201142392416">int toInt(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row1945215360211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p3452123622112">BIGINT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1154166122417">byte[] toBytes(long val)</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p1415418682418">long toLong(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row194521236172116"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1645219360213">FLOAT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p109061555234">byte[] toBytes(float val)</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p29061155172318">float toFloat(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row34521036162118"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p4452193617217">DOUBLE</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1560994310231">byte[] toBytes(double val)</p>
<p id="dli_08_0396__en-us_topic_0000001310215789_p1060919435236">double toDouble(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row245213365217"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1545263612118">DATE</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p18452173612216">Stores the number of days since epoch as an int value.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row1245215363215"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p134521336202116">TIME</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p845216364217">Stores the number of milliseconds of the day as an int value.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row1445210368211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p5452173632110">TIMESTAMP</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p7452103652111">Stores the milliseconds since epoch as a long value.</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row124521366214"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p84521436122116">ARRAY</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p1645233692113">Not supported</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row174521936172119"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p9452193611215">MAP / MULTISET</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p4452113616213">Not supported</p>
</td>
</tr>
<tr id="dli_08_0396__en-us_topic_0000001310215789_row174521363211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0396__en-us_topic_0000001310215789_p18452123618211">ROW</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0396__en-us_topic_0000001310215789_p8452203682114">Not supported</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0396__en-us_topic_0000001310215789_section20985547125411"><h4 class="sectiontitle">Example</h4><p id="dli_08_0396__en-us_topic_0000001310215789_p11305132402910">In this example, data is read from the Kafka data source and written to the HBase result table. The procedure is as follows (the HBase versions used in this example are 1.3.1 and 2.2.3):</p>
<ol id="dli_08_0396__en-us_topic_0000001310215789_ol14211917143211"><li id="dli_08_0396__en-us_topic_0000001310215789_li6127741153217">Create an enhanced datasource connection in the VPC and subnet where HBase and Kafka locate, and bind the connection to the required Flink elastic resource pool. .</li><li id="dli_08_0396__en-us_topic_0000001310215789_li1233002211514">Set HBase and Kafka security groups and add inbound rules to allow access from the Flink queue. Test the connectivity using the HBase and Kafka address. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</li><li id="dli_08_0396__en-us_topic_0000001310215789_li4211017133210">Use the HBase shell to create HBase table <strong id="dli_08_0396__en-us_topic_0000001310215789_b1536010101522">order</strong> that has only one column family <strong id="dli_08_0396__en-us_topic_0000001310215789_b236771013524">detail</strong>.<pre class="screen" id="dli_08_0396__en-us_topic_0000001310215789_screen46881014103714">create 'order', {NAME =&gt; 'detail'}</pre>
</li><li id="dli_08_0396__en-us_topic_0000001310215789_li82171783215">Create a Flink OpenSource SQL job. Enter the following job script and submit the job. The job script uses Kafka as the data source and HBase as the result table (the Rowkey is <strong id="dli_08_0396__en-us_topic_0000001310215789_b165871639135319">order_id</strong> and the column family name is <strong id="dli_08_0396__en-us_topic_0000001310215789_b14633451165319">detail</strong>).<div class="p" id="dli_08_0396__en-us_topic_0000001310215789_p623882218414">When you create a job, set <strong id="dli_08_0396__en-us_topic_0000001310215789_b1037519304565">Flink Version</strong> to <strong id="dli_08_0396__en-us_topic_0000001310215789_b33765309564">1.12</strong> on the <strong id="dli_08_0396__en-us_topic_0000001310215789_b20377163016562">Running Parameters</strong> tab. Select <strong id="dli_08_0396__en-us_topic_0000001310215789_b183771930195618">Save Job Log</strong>, and specify the OBS bucket for saving job logs. <strong id="dli_08_0396__en-us_topic_0000001310215789_b5288135105610">Change the values of the parameters in bold as needed in the following script.</strong><pre class="screen" id="dli_08_0396__en-us_topic_0000001310215789_screen12464143953515">CREATE TABLE orders (
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string
) WITH (
'connector' = 'kafka',
'topic' = '<em id="dli_08_0396__en-us_topic_0000001310215789_i4543155265320"><strong id="dli_08_0396__en-us_topic_0000001310215789_b1754316528535">Kafka</strong></em><em id="dli_08_0396__en-us_topic_0000001310215789_i15341759195319"><strong id="dli_08_0396__en-us_topic_0000001310215789_b5227105913538">Topic</strong></em>',
'properties.bootstrap.servers' = '<em id="dli_08_0396__en-us_topic_0000001310215789_i9319103215313"><strong id="dli_08_0396__en-us_topic_0000001310215789_b0322115133213">KafkaAddress1:KafkaPort,KafkaAddress2:KafkaPort</strong></em>',
'properties.group.id' = '<em id="dli_08_0396__en-us_topic_0000001310215789_i14535174011311"><strong id="dli_08_0396__en-us_topic_0000001310215789_b1466610178329">GroupId</strong></em>',
'scan.startup.mode' = 'latest-offset',
'format' = 'json'
);
create table hbaseSink(
order_id string,
detail Row(
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string)
) with (
'connector' = 'hbase-2.2',
'table-name' = 'order',
'zookeeper.quorum' = '<em id="dli_08_0396__en-us_topic_0000001310215789_i89889572365"><strong id="dli_08_0396__en-us_topic_0000001310215789_b672255714368">ZookeeperAddress:ZookeeperPort</strong></em>',
'sink.buffer-flush.max-rows' = '1'
);
insert into hbaseSink select order_id, Row(order_channel,order_time,pay_amount,real_pay,pay_time,user_id,user_name,area_id) from orders;</pre>
</div>
</li><li id="dli_08_0396__en-us_topic_0000001310215789_li12116173329">Connect to the Kafka cluster and enter the following data to Kafka:<pre class="screen" id="dli_08_0396__en-us_topic_0000001310215789_screen0903104526">{"order_id":"202103241000000001", "order_channel":"webShop", "order_time":"2021-03-24 10:00:00", "pay_amount":"100.00", "real_pay":"100.00", "pay_time":"2021-03-24 10:02:03", "user_id":"0001", "user_name":"Alice", "area_id":"330106"}
{"order_id":"202103241606060001", "order_channel":"appShop", "order_time":"2021-03-24 16:06:06", "pay_amount":"200.00", "real_pay":"180.00", "pay_time":"2021-03-24 16:10:06", "user_id":"0001", "user_name":"Alice", "area_id":"330106"}
{"order_id":"202103251202020001", "order_channel":"miniAppShop", "order_time":"2021-03-25 12:02:02", "pay_amount":"60.00", "real_pay":"60.00", "pay_time":"2021-03-25 12:03:00", "user_id":"0002", "user_name":"Bob", "area_id":"330110"}</pre>
</li><li id="dli_08_0396__en-us_topic_0000001310215789_li15213177329">Run the following statement on the HBase shell to view the data result:<pre class="screen" id="dli_08_0396__en-us_topic_0000001310215789_screen15231525636"> scan 'order'</pre>
<div class="p" id="dli_08_0396__en-us_topic_0000001310215789_p146493283375">The data result is as follows:<pre class="screen" id="dli_08_0396__en-us_topic_0000001310215789_screen122515377317">202103241000000001 column=detail:area_id, timestamp=2021-12-16T21:30:37.954, value=330106
202103241000000001 column=detail:order_channel, timestamp=2021-12-16T21:30:37.954, value=webShop
202103241000000001 column=detail:order_time, timestamp=2021-12-16T21:30:37.954, value=2021-03-24 10:00:00
202103241000000001 column=detail:pay_amount, timestamp=2021-12-16T21:30:37.954, value=@Y\x00\x00\x00\x00\x00\x00
202103241000000001 column=detail:pay_time, timestamp=2021-12-16T21:30:37.954, value=2021-03-24 10:02:03
202103241000000001 column=detail:real_pay, timestamp=2021-12-16T21:30:37.954, value=@Y\x00\x00\x00\x00\x00\x00
202103241000000001 column=detail:user_id, timestamp=2021-12-16T21:30:37.954, value=0001
202103241000000001 column=detail:user_name, timestamp=2021-12-16T21:30:37.954, value=Alice
202103241606060001 column=detail:area_id, timestamp=2021-12-16T21:30:44.842, value=330106
202103241606060001 column=detail:order_channel, timestamp=2021-12-16T21:30:44.842, value=appShop
202103241606060001 column=detail:order_time, timestamp=2021-12-16T21:30:44.842, value=2021-03-24 16:06:06
202103241606060001 column=detail:pay_amount, timestamp=2021-12-16T21:30:44.842, value=@i\x00\x00\x00\x00\x00\x00
202103241606060001 column=detail:pay_time, timestamp=2021-12-16T21:30:44.842, value=2021-03-24 16:10:06
202103241606060001 column=detail:real_pay, timestamp=2021-12-16T21:30:44.842, value=@f\x80\x00\x00\x00\x00\x00
202103241606060001 column=detail:user_id, timestamp=2021-12-16T21:30:44.842, value=0001
202103241606060001 column=detail:user_name, timestamp=2021-12-16T21:30:44.842, value=Alice
202103251202020001 column=detail:area_id, timestamp=2021-12-16T21:30:52.181, value=330110
202103251202020001 column=detail:order_channel, timestamp=2021-12-16T21:30:52.181, value=miniAppShop
202103251202020001 column=detail:order_time, timestamp=2021-12-16T21:30:52.181, value=2021-03-25 12:02:02
202103251202020001 column=detail:pay_amount, timestamp=2021-12-16T21:30:52.181, value=@N\x00\x00\x00\x00\x00\x00
202103251202020001 column=detail:pay_time, timestamp=2021-12-16T21:30:52.181, value=2021-03-25 12:03:00
202103251202020001 column=detail:real_pay, timestamp=2021-12-16T21:30:52.181, value=@N\x00\x00\x00\x00\x00\x00
202103251202020001 column=detail:user_id, timestamp=2021-12-16T21:30:52.181, value=0002
202103251202020001 column=detail:user_name, timestamp=2021-12-16T21:30:52.181, value=Bob</pre>
</div>
</li></ol>
</div>
<div class="section" id="dli_08_0396__en-us_topic_0000001310215789_section15293124454517"><h4 class="sectiontitle">FAQ</h4><p id="dli_08_0396__en-us_topic_0000001310215789_p271112372118">Q: What should I do if the Flink job execution fails and the log contains the following error information?</p>
<pre class="screen" id="dli_08_0396__en-us_topic_0000001310215789_screen3298310104813">org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 90069ms for connection id 0x0</pre>
<p id="dli_08_0396__en-us_topic_0000001310215789_p1526017594146">A: The datasource connection is not bound or the binding fails. Configure the datasource connection or configure the security group of the Kafka cluster to allow access from the DLI queue.</p>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0391.html">Creating Result Tables</a></div>
</div>
</div>
View Git Blame Copy Permalink