vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0404.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

338 lines
33 KiB
HTML
Raw Blame History

<a name="dli_08_0404"></a><a name="dli_08_0404"></a>
<h1 class="topictitle1">HBase Dimension Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0404__en-us_topic_0000001310215793_section102271660319"><h4 class="sectiontitle">Function</h4><p id="dli_08_0404__en-us_topic_0000001310215793_p28331612932">Create a Hbase dimension table to connect to the source streams for wide table generation.</p>
</div>
<div class="section" id="dli_08_0404__en-us_topic_0000001310215793_section5611103820312"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0404__en-us_topic_0000001310215793_ul66761330172912"><li id="dli_08_0404__en-us_topic_0000001310215793_li13676330102913">An enhanced datasource connection has been created for DLI to connect to HBase, so that jobs can run on the dedicated queue of DLI and you can set the security group rules as required.
</li><li id="dli_08_0404__en-us_topic_0000001310215793_li1380793232118">If MRS HBase is used, IP addresses of all hosts in the MRS cluster have been added to host information of the enhanced datasource connection.<p id="dli_08_0404__en-us_topic_0000001310215793_p1679341219"><a name="dli_08_0404__en-us_topic_0000001310215793_li1380793232118"></a><a name="en-us_topic_0000001310215793_li1380793232118"></a>.</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p1290940152113">For details, see section "Modifying the Host Information" in the <em id="dli_08_0404__en-us_topic_0000001310215793_i22712221019">Data Lake Insight User Guide</em>.</p>
</li></ul>
</div>
<div class="section" id="dli_08_0404__en-us_topic_0000001310215793_section1465831815212"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0404__en-us_topic_0000001310215793_ul322397111719"><li id="dli_08_0404__en-us_topic_0000001310215793_li2703203462516">When you create a Flink OpenSource SQL job, set <strong id="dli_08_0404__en-us_topic_0000001310215793_b187506461018">Flink Version</strong> to <strong id="dli_08_0404__en-us_topic_0000001310215793_b77509431017">1.12</strong> in the <strong id="dli_08_0404__en-us_topic_0000001310215793_b47513451019">Running Parameters</strong> tab. Select <strong id="dli_08_0404__en-us_topic_0000001310215793_b1175110414107">Save Job Log</strong>, and specify the OBS bucket for saving job logs.</li><li id="dli_08_0404__en-us_topic_0000001310215793_li167143475422">All the column families in HBase table must be declared as ROW type, the field name maps to the column family name, and the nested field names map to the column qualifier names. There is no need to declare all the families and qualifiers in the schema, users can declare what is used in the query. Except the ROW type fields, the single atomic type field (for example, STRING, BIGINT) will be recognized as HBase rowkey. The rowkey field can be an arbitrary name, but should be quoted using backticks if it is a reserved keyword.</li></ul>
</div>
<div class="section" id="dli_08_0404__en-us_topic_0000001310215793_section661775715318"><h4 class="sectiontitle">Syntax</h4><pre class="screen" id="dli_08_0404__en-us_topic_0000001310215793_screen1214311454414">create table hbaseSource (
attr_name attr_type
(',' attr_name attr_type)*
)
with (
'connector' = 'hbase-2.2',
'table-name' = '',
'zookeeper.quorum' = ''
);</pre>
</div>
<div class="section" id="dli_08_0404__en-us_topic_0000001310215793_section4712115614410"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0404__en-us_topic_0000001310215793_table197671281054" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0404__en-us_topic_0000001310215793_row15767728752"><th align="left" class="cellrowborder" valign="top" width="23.24%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0404__en-us_topic_0000001310215793_p11767182814517">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.35%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0404__en-us_topic_0000001310215793_p18767928456">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="8.12%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0404__en-us_topic_0000001310215793_p157311453106">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="11.01%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0404__en-us_topic_0000001310215793_p651315488103">Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="48.28%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0404__en-us_topic_0000001310215793_p67675288514">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0404__en-us_topic_0000001310215793_row77676281454"><td class="cellrowborder" valign="top" width="23.24%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1476719281051">connector</p>
</td>
<td class="cellrowborder" valign="top" width="9.35%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p37676282054">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="8.12%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0404__en-us_topic_0000001310215793_p3573645131015">None</p>
</td>
<td class="cellrowborder" valign="top" width="11.01%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0404__en-us_topic_0000001310215793_p9513124811019">String</p>
</td>
<td class="cellrowborder" valign="top" width="48.28%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1076715284510">Connector type. Set this parameter to <strong id="dli_08_0404__en-us_topic_0000001310215793_b724191719173">hbase-2.2</strong>.</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row8767192815513"><td class="cellrowborder" valign="top" width="23.24%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p07673281152">table-name</p>
</td>
<td class="cellrowborder" valign="top" width="9.35%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p14767028656">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="8.12%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1239515461648">None</p>
</td>
<td class="cellrowborder" valign="top" width="11.01%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0404__en-us_topic_0000001310215793_p351394831011">String</p>
</td>
<td class="cellrowborder" valign="top" width="48.28%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0404__en-us_topic_0000001310215793_p94571843121213">Name of the HBase table</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row2076712810512"><td class="cellrowborder" valign="top" width="23.24%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p3767428555">zookeeper.quorum</p>
</td>
<td class="cellrowborder" valign="top" width="9.35%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p076717282053">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="8.12%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0404__en-us_topic_0000001310215793_p165735455108">None</p>
</td>
<td class="cellrowborder" valign="top" width="11.01%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0404__en-us_topic_0000001310215793_p13513184815101">String</p>
</td>
<td class="cellrowborder" valign="top" width="48.28%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0404__en-us_topic_0000001310215793_p17672288517">HBase Zookeeper quorum. The format is ZookeeperAddress:ZookeeperPort.</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p9871653181210">The following describes how to obtain the ZooKeeper IP address and port number:</p>
<ul id="dli_08_0404__en-us_topic_0000001310215793_ul571954114435"><li id="dli_08_0404__en-us_topic_0000001310215793_li177201341154318">On the MRS Manager console, choose <strong id="dli_08_0404__en-us_topic_0000001310215793_b10728101562216">Cluster</strong> &gt; <em id="dli_08_0404__en-us_topic_0000001310215793_i5729161514227">Name of the desired cluster</em> &gt; <strong id="dli_08_0404__en-us_topic_0000001310215793_b1873021515221">Service</strong> &gt; <strong id="dli_08_0404__en-us_topic_0000001310215793_b117312015192213">ZooKeeper</strong> &gt; <strong id="dli_08_0404__en-us_topic_0000001310215793_b37311015152216">Instance</strong>. On the displayed page, obtain the IP address of the ZooKeeper instance.</li><li id="dli_08_0404__en-us_topic_0000001310215793_li1433443194310">On the MRS Manager console, choose <strong id="dli_08_0404__en-us_topic_0000001310215793_b14816112112315">Cluster</strong> &gt; <em id="dli_08_0404__en-us_topic_0000001310215793_i1747361019236">Name of the desired cluster</em> &gt; <strong id="dli_08_0404__en-us_topic_0000001310215793_b5745131432312">Service</strong> &gt; <strong id="dli_08_0404__en-us_topic_0000001310215793_b187151617162313">ZooKeeper</strong> &gt; <strong id="dli_08_0404__en-us_topic_0000001310215793_b641232113238">Configuration</strong>, and click <strong id="dli_08_0404__en-us_topic_0000001310215793_b810571314246">All Configurations</strong>. Search for the <strong id="dli_08_0404__en-us_topic_0000001310215793_b0287204212240">clientPort</strong> parameter, and obtain the ZooKeeper port number.</li></ul>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row137672281756"><td class="cellrowborder" valign="top" width="23.24%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p12767228856">zookeeper.znode.parent</p>
</td>
<td class="cellrowborder" valign="top" width="9.35%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p77677281518">No</p>
</td>
<td class="cellrowborder" valign="top" width="8.12%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0404__en-us_topic_0000001310215793_p116661237141320">/hbase</p>
</td>
<td class="cellrowborder" valign="top" width="11.01%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0404__en-us_topic_0000001310215793_p135132480108">String</p>
</td>
<td class="cellrowborder" valign="top" width="48.28%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1971655131210">Root directory in ZooKeeper for the HBase cluster.</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row122139111148"><td class="cellrowborder" valign="top" width="23.24%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p16214141115149">lookup.async</p>
</td>
<td class="cellrowborder" valign="top" width="9.35%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p12141711181416">No</p>
</td>
<td class="cellrowborder" valign="top" width="8.12%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0404__en-us_topic_0000001310215793_p4214131113145">false</p>
</td>
<td class="cellrowborder" valign="top" width="11.01%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0404__en-us_topic_0000001310215793_p7214171118146">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="48.28%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0404__en-us_topic_0000001310215793_p4214141117145">Whether async lookup is enabled.</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row1224620148147"><td class="cellrowborder" valign="top" width="23.24%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p42461814121418">lookup.cache.max-rows</p>
</td>
<td class="cellrowborder" valign="top" width="9.35%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p102461614141411">No</p>
</td>
<td class="cellrowborder" valign="top" width="8.12%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0404__en-us_topic_0000001310215793_p19246151416142">-1</p>
</td>
<td class="cellrowborder" valign="top" width="11.01%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0404__en-us_topic_0000001310215793_p192461814191411">Long</p>
</td>
<td class="cellrowborder" valign="top" width="48.28%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0404__en-us_topic_0000001310215793_p16266183413234">The max number of rows of lookup cache. Caches exceeding the TTL will be expired.</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p17903183613314">Lookup cache is disabled by default.</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row7677101614146"><td class="cellrowborder" valign="top" width="23.24%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p10678916131411">lookup.cache.ttl</p>
</td>
<td class="cellrowborder" valign="top" width="9.35%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p567813168148">No</p>
</td>
<td class="cellrowborder" valign="top" width="8.12%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0404__en-us_topic_0000001310215793_p06786169145">-1</p>
</td>
<td class="cellrowborder" valign="top" width="11.01%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0404__en-us_topic_0000001310215793_p8678121681415">Long</p>
</td>
<td class="cellrowborder" valign="top" width="48.28%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0404__en-us_topic_0000001310215793_p526611342233">Maximum time to live (TTL) of for every rows in lookup cache. Caches exceeding the TTL will be expired. The format is {length value}{time unit label}, for example, <strong id="dli_08_0404__en-us_topic_0000001310215793_b25395142813">123ms, 321s</strong>. The supported time units include d, h, min, s, and ms (default unit).</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p9272323163315">Lookup cache is disabled by default.</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row182151044161711"><td class="cellrowborder" valign="top" width="23.24%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1421504431712">lookup.max-retries</p>
</td>
<td class="cellrowborder" valign="top" width="9.35%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p2021564471715">No</p>
</td>
<td class="cellrowborder" valign="top" width="8.12%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0404__en-us_topic_0000001310215793_p4215134415178">3</p>
</td>
<td class="cellrowborder" valign="top" width="11.01%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0404__en-us_topic_0000001310215793_p6215144161718">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="48.28%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0404__en-us_topic_0000001310215793_p3215164415179">Maximum retry times if lookup database failed.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0404__en-us_topic_0000001310215793_section11151124192013"><h4 class="sectiontitle">Data Type Mapping</h4><p id="dli_08_0404__en-us_topic_0000001310215793_p169933132115">HBase stores all data as byte arrays. The data needs to be serialized and deserialized during read and write operation.</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p66992342118">When serializing and de-serializing, Flink HBase connector uses utility class <strong id="dli_08_0404__en-us_topic_0000001310215793_b2122586299">org.apache.hadoop.hbase.util.Bytes</strong> provided by HBase (Hadoop) to convert Flink data types to and from byte arrays.</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p269919319215">Flink HBase connector encodes null values to empty bytes, and decode empty bytes to null values for all data types except string type. For string type, the null literal is determined by null-string-literal option.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0404__en-us_topic_0000001310215793_table17452636132111" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Data type mapping</caption><thead align="left"><tr id="dli_08_0404__en-us_topic_0000001310215793_row1345263617217"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.6.5.2.3.1.1"><p id="dli_08_0404__en-us_topic_0000001310215793_p16452436132117">Flink SQL Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.6.5.2.3.1.2"><p id="dli_08_0404__en-us_topic_0000001310215793_p12452123662114">HBase Conversion</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0404__en-us_topic_0000001310215793_row14452636132115"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p2045273672116">CHAR / VARCHAR / STRING</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p427912322519">byte[] toBytes(String s)</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p12279192302515">String toString(byte[] b)</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row1145216365211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p19452143632119">BOOLEAN</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p759018125250">byte[] toBytes(boolean b)</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p4590212192515">boolean toBoolean(byte[] b)</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row17452153612211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p18452103611212">BINARY / VARBINARY</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p14452736182115">Return byte[] as is.</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row34521365219"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p13452173612116">DECIMAL</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1214719587243">byte[] toBytes(BigDecimal v)</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p614745818244">BigDecimal toBigDecimal(byte[] b)</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row194521636162114"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p94521636202112">TINYINT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1211974622411">new byte[] { val }</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p1119174610240">bytes[0] // returns first and only byte from bytes</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row1452183613212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1452133616212">SMALLINT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1454183510245">byte[] toBytes(short val)</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p17541435142410">short toShort(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row445243617216"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p2452636112110">INT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1320111234249">byte[] toBytes(int val)</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p18201142392416">int toInt(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row1945215360211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p3452123622112">BIGINT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1154166122417">byte[] toBytes(long val)</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p1415418682418">long toLong(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row194521236172116"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1645219360213">FLOAT</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p109061555234">byte[] toBytes(float val)</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p29061155172318">float toFloat(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row34521036162118"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p4452193617217">DOUBLE</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1560994310231">byte[] toBytes(double val)</p>
<p id="dli_08_0404__en-us_topic_0000001310215793_p1060919435236">double toDouble(byte[] bytes)</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row245213365217"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1545263612118">DATE</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p18452173612216">Number of days since 1970-01-01 00:00:00 UTC. The value is an integer.</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row1245215363215"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p134521336202116">TIME</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p845216364217">Number of milliseconds since 1970-01-01 00:00:00 UTC. The value is an integer.</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row1445210368211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p5452173632110">TIMESTAMP</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p7452103652111">Number of milliseconds since 1970-01-01 00:00:00 UTC. The value is of the long type.</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row124521366214"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p84521436122116">ARRAY</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p1645233692113">Not supported</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row174521936172119"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p9452193611215">MAP / MULTISET</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p4452113616213">Not supported</p>
</td>
</tr>
<tr id="dli_08_0404__en-us_topic_0000001310215793_row174521363211"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.1 "><p id="dli_08_0404__en-us_topic_0000001310215793_p18452123618211">ROW</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.6.5.2.3.1.2 "><p id="dli_08_0404__en-us_topic_0000001310215793_p8452203682114">Not supported</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0404__en-us_topic_0000001310215793_section111313461386"><h4 class="sectiontitle">Example</h4><p id="dli_08_0404__en-us_topic_0000001310215793_p11305132402910">In this example, data is read from a Kafka data source, an HBase table is used as a dimension table to generate a wide table, and the result is written to a Kafka result table. The procedure is as follows (the HBase versions in this example are 1.3.1 and 2.2.3):</p>
<ol id="dli_08_0404__en-us_topic_0000001310215793_ol11671552124818"><li id="dli_08_0404__en-us_topic_0000001310215793_li6127741153217">Create an enhanced datasource connection in the VPC and subnet where HBase and Kafka locate, and bind the connection to the required Flink elastic resource pool. Add MRS host information for the enhanced datasource connection..</li><li id="dli_08_0404__en-us_topic_0000001310215793_li1233002211514">Set HBase and Kafka security groups and add inbound rules to allow access from the Flink queue. Test the connectivity using the HBase and Kafka address. If the connection passes the test, it is bound to the queue.</li><li id="dli_08_0404__en-us_topic_0000001310215793_li146711352194820">Create a HBase table and name it <strong id="dli_08_0404__en-us_topic_0000001310215793_b20692192715598">area_info</strong> using the HBase shell. The table has only one column family <strong id="dli_08_0404__en-us_topic_0000001310215793_b441911820119">detail</strong>. The creation statement is as follows:<pre class="screen" id="dli_08_0404__en-us_topic_0000001310215793_screen46881014103714">create 'area_info', {NAME =&gt; 'detail'}</pre>
</li><li id="dli_08_0404__en-us_topic_0000001310215793_li1567114521480">Run the following statement in the HBase shell to insert dimension table data:<pre class="screen" id="dli_08_0404__en-us_topic_0000001310215793_screen948154010134">put 'area_info', '330106', 'detail:area_province_name', 'a1'
put 'area_info', '330106', 'detail:area_city_name', 'b1'
put 'area_info', '330106', 'detail:area_county_name', 'c2'
put 'area_info', '330106', 'detail:area_street_name', 'd2'
put 'area_info', '330106', 'detail:region_name', 'e1'
put 'area_info', '330110', 'detail:area_province_name', 'a1'
put 'area_info', '330110', 'detail:area_city_name', 'b1'
put 'area_info', '330110', 'detail:area_county_name', 'c4'
put 'area_info', '330110', 'detail:area_street_name', 'd4'
put 'area_info', '330110', 'detail:region_name', 'e1'</pre>
</li><li id="dli_08_0404__en-us_topic_0000001310215793_li8671752194812">Create a Flink OpenSource SQL job Enter the following job script and submit the job. The job script uses Kafka as the data source and an HBase table as the dimension table. Data is output to a Kafka result table.<div class="p" id="dli_08_0404__en-us_topic_0000001310215793_p9653151151512"><a name="dli_08_0404__en-us_topic_0000001310215793_li8671752194812"></a><a name="en-us_topic_0000001310215793_li8671752194812"></a>When you create a job, set <strong id="dli_08_0404__en-us_topic_0000001310215793_b13103754335">Flink Version</strong> to <strong id="dli_08_0404__en-us_topic_0000001310215793_b121031549317">1.12</strong> in the <strong id="dli_08_0404__en-us_topic_0000001310215793_b2104155418318">Running Parameters</strong> tab. Select <strong id="dli_08_0404__en-us_topic_0000001310215793_b810420544315">Save Job Log</strong>, and specify the OBS bucket for saving job logs. <strong id="dli_08_0404__en-us_topic_0000001310215793_b1778395617316">Set the values of the parameters in bold in the following script as needed.</strong><pre class="screen" id="dli_08_0404__en-us_topic_0000001310215793_screen372054861512">CREATE TABLE orders (
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string,
proctime as Proctime()
) WITH (
'connector' = 'kafka',
'topic' = '<em id="dli_08_0404__en-us_topic_0000001310215793_i4543155265320"><strong id="dli_08_0404__en-us_topic_0000001310215793_b1754316528535">Kafka</strong></em><em id="dli_08_0404__en-us_topic_0000001310215793_i15341759195319"><strong id="dli_08_0404__en-us_topic_0000001310215793_b5227105913538">SourceTopic</strong></em>',
'properties.bootstrap.servers' = '<em id="dli_08_0404__en-us_topic_0000001310215793_i9319103215313"><strong id="dli_08_0404__en-us_topic_0000001310215793_b0322115133213">KafkaAddress1:KafkaPort,KafkaAddress2:KafkaPort</strong></em>',
'properties.group.id' = '<em id="dli_08_0404__en-us_topic_0000001310215793_i14535174011311"><strong id="dli_08_0404__en-us_topic_0000001310215793_b1466610178329">GroupId</strong></em>',
'scan.startup.mode' = 'latest-offset',
'format' = 'json'
);
-- Create an address dimension table
create table area_info (
area_id string,
detail row(
area_province_name string,
area_city_name string,
area_county_name string,
area_street_name string,
region_name string)
) WITH (
'connector' = 'hbase-2.2',
'table-name' = 'area_info',
'zookeeper.quorum' = '<em id="dli_08_0404__en-us_topic_0000001310215793_i89889572365"><strong id="dli_08_0404__en-us_topic_0000001310215793_b672255714368">ZookeeperAddress:ZookeeperPort</strong></em>',
'lookup.async' = 'true',
'lookup.cache.max-rows' = '10000',
'lookup.cache.ttl' = '2h'
);
-- Generate a wide table based on the address dimension table containing detailed order information.
create table order_detail(
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id string,
area_province_name string,
area_city_name string,
area_county_name string,
area_street_name string,
region_name string
) with (
'connector' = 'kafka',
'topic' = '&lt;yourSinkTopic&gt;',
'properties.bootstrap.servers' = '<em id="dli_08_0404__en-us_topic_0000001310215793_i176756261176"><strong id="dli_08_0404__en-us_topic_0000001310215793_b867582671710">KafkaAddress1:KafkaPort,KafkaAddress2:KafkaPort</strong></em>',
'format' = 'json'
);
insert into order_detail
select orders.order_id, orders.order_channel, orders.order_time, orders.pay_amount, orders.real_pay, orders.pay_time, orders.user_id, orders.user_name,
area.area_id, area.area_province_name, area.area_city_name, area.area_county_name,
area.area_street_name, area.region_name from orders
left join area_info for system_time as of orders.proctime as area on orders.area_id = area.area_id;</pre>
</div>
</li><li id="dli_08_0404__en-us_topic_0000001310215793_li1167135274817">Connect to the Kafka cluster and insert the following test data into the source topic in Kafka:<pre class="screen" id="dli_08_0404__en-us_topic_0000001310215793_screen0903104526">{"order_id":"202103241000000001", "order_channel":"webShop", "order_time":"2021-03-24 10:00:00", "pay_amount":"100.00", "real_pay":"100.00", "pay_time":"2021-03-24 10:02:03", "user_id":"0001", "user_name":"Alice", "area_id":"330106"}
{"order_id":"202103241606060001", "order_channel":"appShop", "order_time":"2021-03-24 16:06:06", "pay_amount":"200.00", "real_pay":"180.00", "pay_time":"2021-03-24 16:10:06", "user_id":"0001", "user_name":"Alice", "area_id":"330106"}
{"order_id":"202103251202020001", "order_channel":"miniAppShop", "order_time":"2021-03-25 12:02:02", "pay_amount":"60.00", "real_pay":"60.00", "pay_time":"2021-03-25 12:03:00", "user_id":"0002", "user_name":"Bob", "area_id":"330110"}</pre>
</li><li id="dli_08_0404__en-us_topic_0000001310215793_li26712521486">Connect to the Kafka cluster and read data from the sink topic of Kafka. The result data is as follows:<pre class="screen" id="dli_08_0404__en-us_topic_0000001310215793_screen15231525636">{"order_id":"202103241000000001","order_channel":"webShop","order_time":"2021-03-24 10:00:00","pay_amount":100.0,"real_pay":100.0,"pay_time":"2021-03-24 10:02:03","user_id":"0001","user_name":"Alice","area_id":"330106","area_province_name":"a1","area_city_name":"b1","area_county_name":"c2","area_street_name":"d2","region_name":"e1"}
{"order_id":"202103241606060001","order_channel":"appShop","order_time":"2021-03-24 16:06:06","pay_amount":200.0,"real_pay":180.0,"pay_time":"2021-03-24 16:10:06","user_id":"0001","user_name":"Alice","area_id":"330106","area_province_name":"a1","area_city_name":"b1","area_county_name":"c2","area_street_name":"d2","region_name":"e1"}
{"order_id":"202103251202020001","order_channel":"miniAppShop","order_time":"2021-03-25 12:02:02","pay_amount":60.0,"real_pay":60.0,"pay_time":"2021-03-25 12:03:00","user_id":"0002","user_name":"Bob","area_id":"330110","area_province_name":"a1","area_city_name":"b1","area_county_name":"c4","area_street_name":"d4","region_name":"e1"}</pre>
</li></ol>
</div>
<div class="section" id="dli_08_0404__en-us_topic_0000001310215793_section1473475361918"><h4 class="sectiontitle">FAQs</h4><p id="dli_08_0404__en-us_topic_0000001310215793_p198183356371">Q: What should I do if Flink job logs contain the following error information?</p>
<pre class="screen" id="dli_08_0404__en-us_topic_0000001310215793_screen3298310104813">org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 90069ms for connection id 0x0</pre>
<p id="dli_08_0404__en-us_topic_0000001310215793_p1526017594146">A: The datasource connection is not bound or the binding fails. Configure the datasource connection or configure the security group of the Kafka cluster to allow access from the DLI queue.</p>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0402.html">Creating Dimension Tables</a></div>
</div>
</div>
View Git Blame Copy Permalink