forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
94 lines
13 KiB
HTML
94 lines
13 KiB
HTML
<a name="mrs_01_1128"></a><a name="mrs_01_1128"></a>
|
|
|
|
<h1 class="topictitle1">Hive input</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_1128__en-us_topic_0000001173470776_s1757769f40ce4b56b8762ed81bdcf93b"><h4 class="sectiontitle">Overview</h4><p id="mrs_01_1128__en-us_topic_0000001173470776_a565ad3678cdc4edf8d689cf299c4fdb5">The <span class="parmname" id="mrs_01_1128__en-us_topic_0000001173470776_p8bd557c482784b6cbfb55ee51b53c99c"><b>Hive Input</b></span> operator converts specified columns in an HBase table into input fields of the same quantity.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1128__en-us_topic_0000001173470776_s6d0a4e5789d7444aa2f93f36b38baa80"><h4 class="sectiontitle">Input and Output</h4><ul id="mrs_01_1128__en-us_topic_0000001173470776_u5df1c265be714b0aaccc85b0312c5a22"><li id="mrs_01_1128__en-us_topic_0000001173470776_l22cc09f87daa493ea0d6aeb9ebfcab94">Input: Hive table columns</li><li id="mrs_01_1128__en-us_topic_0000001173470776_l14066072556944b59019808a9cefab14">Output: fields</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1128__en-us_topic_0000001173470776_s997925e754c04f86b10ad505e2c7b8b6"><h4 class="sectiontitle">Parameters</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1128__en-us_topic_0000001173470776_tbe6be3b0961048da9a8a4db19908086a" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Operator parameters description</caption><thead align="left"><tr id="mrs_01_1128__en-us_topic_0000001173470776_r7ff753487d104349a0a144ecdb0b2dc6"><th align="left" class="cellrowborder" valign="top" width="18.37%" id="mcps1.3.3.2.2.6.1.1"><p id="mrs_01_1128__en-us_topic_0000001173470776_ae420d7eb950341e3b3dca084cc082e4d">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="54.730000000000004%" id="mcps1.3.3.2.2.6.1.2"><p id="mrs_01_1128__en-us_topic_0000001173470776_a6855ed06507447e4bf198374d63627ed">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="7.5200000000000005%" id="mcps1.3.3.2.2.6.1.3"><p id="mrs_01_1128__en-us_topic_0000001173470776_a02ae3b037885491fa6e0525c670d9e5c">Node Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="9.69%" id="mcps1.3.3.2.2.6.1.4"><p id="mrs_01_1128__en-us_topic_0000001173470776_a8d986fefa57e48258204f08c03a149f8">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="9.69%" id="mcps1.3.3.2.2.6.1.5"><p id="mrs_01_1128__en-us_topic_0000001173470776_af0f6c4b17f294c6f93ced116295890f8">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1128__en-us_topic_0000001173470776_rd16e392d57134958ae9142041c620511"><td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.3.2.2.6.1.1 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a8e92a533a6c44b3e9526b0cf715e02bf">Hive database</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="54.730000000000004%" headers="mcps1.3.3.2.2.6.1.2 "><p id="mrs_01_1128__en-us_topic_0000001173470776_ae59d300b450444c38fb1f64c9a66147a">Name of a Hive database</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.2.2.6.1.3 "><p id="mrs_01_1128__en-us_topic_0000001173470776_ab861feea3bd143309193a89525306044">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.4 "><p id="mrs_01_1128__en-us_topic_0000001173470776_aa835eac6d3df4d9fabeaedce30a9448c">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.5 "><p id="mrs_01_1128__en-us_topic_0000001173470776_af9a8d8adbba244c984890bb5c25e9060">default</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1128__en-us_topic_0000001173470776_rc1984899782b4701b4baccc428e28703"><td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.3.2.2.6.1.1 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a6e14b39cf04f483ea79052ff1891fe9b">Hive table name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="54.730000000000004%" headers="mcps1.3.3.2.2.6.1.2 "><p id="mrs_01_1128__en-us_topic_0000001173470776_ab5728c1ab6b84fd0a653067929087870">Name of the Hive table configured</p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_a2776939c22754abb8e211e52cbc4ca54">Only one Hive table is supported.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.2.2.6.1.3 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a0c59400e6dfb4e679094611043282972">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.4 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a95e4ba0498174f4bab2af86c70173021">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.5 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a4ebf15f6bb5f4cedb4989a546a6155a9">None.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1128__en-us_topic_0000001173470776_r601a47ac6f3d4526b359188e1dbcf511"><td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.3.2.2.6.1.1 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a848bc021b584430ebddd49cbb8f7637b">Partition filter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="54.730000000000004%" headers="mcps1.3.3.2.2.6.1.2 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a99d5d6410f2048ce87e35226d327dab0">Configures the partition filter can export data of specific partitions. The parameter is null by default and data of the whole table can be exported.</p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_a26f8159bb74a4da2b09ec920ee3280a2">For example, to export data of a table whose partition field's locale value is <span class="parmvalue" id="mrs_01_1128__en-us_topic_0000001173470776_parmvalue100149507791814"><b>CN</b></span> or <span class="parmvalue" id="mrs_01_1128__en-us_topic_0000001173470776_parmvalue191930835291814"><b>US</b></span>, the input is as follows:</p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_a89d3d21637f944e49cc5f5d3828e0074"><strong id="mrs_01_1128__en-us_topic_0000001173470776_a0400aaf98b3a479fa9b2347451802c97">locale = "CN" or locale = "US"</strong></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.2.2.6.1.3 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a12032039fca24ed4a56c512a16d494ee">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.4 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a544f5047354a48e8b35920a8442c3ccc">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.5 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a699fd1dd78e248d48d24553a3f4ae729">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1128__en-us_topic_0000001173470776_ra43afe87519f4b00b9fc282697d3adac"><td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.3.2.2.6.1.1 "><p id="mrs_01_1128__en-us_topic_0000001173470776_ada265173b6374aa6b7ae73f75776d497">Hive input field</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="54.730000000000004%" headers="mcps1.3.3.2.2.6.1.2 "><p id="mrs_01_1128__en-us_topic_0000001173470776_abe26c1296e4e4eb5b664ce8525405478">Configures the input information of Hive</p>
|
|
<ul id="mrs_01_1128__en-us_topic_0000001173470776_u66c32f40ce47470ea2537099a829a5fa"><li id="mrs_01_1128__en-us_topic_0000001173470776_l289c507731c6454d8a7bfb04e63e7d9c">column name: Hive column name.</li><li id="mrs_01_1128__en-us_topic_0000001173470776_l7ecc5b75ca984022998da91817fa3ce3">field name: Input field name.</li><li id="mrs_01_1128__en-us_topic_0000001173470776_l36d58c55bc83461588b0fa99a8f40679">type: Field type.</li><li id="mrs_01_1128__en-us_topic_0000001173470776_lcafda33849634c4087fe4835230188e5">length: Field value length. If the actual field value is excessively long, the value is cut based on the configured length. When <span class="parmname" id="mrs_01_1128__en-us_topic_0000001173470776_parmname56492572291814"><b>type</b></span> is set to <span class="parmvalue" id="mrs_01_1128__en-us_topic_0000001173470776_p71f5c05819a04c93ba84da69a2d5df7f"><b>CHAR</b></span>, spaces are added to the field value for supplement if the actual field value length is less than the configured length. When <span class="parmname" id="mrs_01_1128__en-us_topic_0000001173470776_parmname179289049891814"><b>type</b></span> is set to <span class="parmvalue" id="mrs_01_1128__en-us_topic_0000001173470776_p679ac09ed6e440168cdcb85e8d962715"><b>VARCHAR</b></span>, no space is added to the field value for supplement if the actual field value length is less than the configured length.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.2.2.6.1.3 "><p id="mrs_01_1128__en-us_topic_0000001173470776_a0521d6d895ca4cdab646a9d32600b37f">map</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.4 "><p id="mrs_01_1128__en-us_topic_0000001173470776_aa70a7ca0e7f541deacd08e427a8bc8cc">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.5 "><p id="mrs_01_1128__en-us_topic_0000001173470776_ac0b13eb81f50449b96cf05abd85c3db9">-</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1128__en-us_topic_0000001173470776_s69f50729148e40ad92ba03e8d42d54ca"><h4 class="sectiontitle">Data Processing Rule</h4><ul id="mrs_01_1128__en-us_topic_0000001173470776_ud0c72e3677674863a799d976c4decede"><li id="mrs_01_1128__en-us_topic_0000001173470776_l849153f6f231442f9d1450ea7fb2f2bf">If the Hive table name does not exist, the job fails to be submitted.</li><li id="mrs_01_1128__en-us_topic_0000001173470776_led31af6014ad4deaa373c2ce1534fdf1">If the configured column names are inconsistent with the Hive table column names, the data cannot be read and the number of imported data records is 0.</li><li id="mrs_01_1128__en-us_topic_0000001173470776_l1313bf620c0e40909e386b52e9c6a6d8">If the field value does not match the actual type, the data in the line will become dirty data.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1128__en-us_topic_0000001173470776_s436a81c6b27c4704a679885922c7e9bf"><h4 class="sectiontitle">Example</h4><p id="mrs_01_1128__en-us_topic_0000001173470776_aa4f7fc9893674bc198b5fa6451b2797f">Use the data export from Hive to SQL Server 2014 as an example.</p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_a669856820d3148a48fb34301d1813701">In SQL Server 2014, run the following statement to create an empty table <span class="parmname" id="mrs_01_1128__en-us_topic_0000001173470776_parmname83599833091814"><b>test_1</b></span> for storing Hive data. Run the following statement:</p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_a3288e1b7aeb34673a7e72e816a77f939"><strong id="mrs_01_1128__en-us_topic_0000001173470776_af9ef2e5f66f8451e8ebc45be40681e4c">create table test_1 (id int, name text, value text);</strong></p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_acc35663e0ff243fa920dcff5bf08a446">Configure the <span class="parmname" id="mrs_01_1128__en-us_topic_0000001173470776_p766c9367c82b440182edc8658f7697e3"><b>Hive Input</b></span> operator to generate fields A, B, and C.</p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_afee5b00f675f4562b20e6caf33ddcd6d">After the data connector is set, click <span class="menucascade" id="mrs_01_1128__en-us_topic_0000001173470776_menucascade201988924991814"><b><span class="uicontrol" id="mrs_01_1128__en-us_topic_0000001173470776_uicontrol45913520791814">Automatic Identification</span></b></span>. The system will automatically read fields in the database and select required fields for adding. You only need to optimize or modify the fields manually based on service scenarios.</p>
|
|
<div class="note" id="mrs_01_1128__en-us_topic_0000001173470776_n23d5068d3871453490389527d113b376"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1128__en-us_topic_0000001173470776_a3729c18fbed149c2818ac65245126b60">Performing this operation will overwrite existing data in the table.</p>
|
|
</div></div>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_afdcf531457c8471fbd679dd78c99de1e"><span><img id="mrs_01_1128__en-us_topic_0000001173470776_i09f45fa8b76d4eecae63779032432eff" src="en-us_image_0000001349259305.png"></span></p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_a9e36143ea9764f979c8765ce2ab5a85c">Use the <span class="parmname" id="mrs_01_1128__en-us_topic_0000001173470776_parmname129463082991814"><b>Table Out</b></span> operator to export A, B, and C to the <span class="parmname" id="mrs_01_1128__en-us_topic_0000001173470776_parmname23448646991814"><b>test_1</b></span> table.</p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_adc1b4fa7722c4cdf90cffd087d480261"><strong id="mrs_01_1128__en-us_topic_0000001173470776_aba6239ccdaf04c5696d2f9f1709f60e7">select * from test_1;</strong></p>
|
|
<p id="mrs_01_1128__en-us_topic_0000001173470776_af82df34bc7254f6bb3378b9630198ed3"><span><img id="mrs_01_1128__en-us_topic_0000001173470776_ie40a04420bfd487e9f87a008c609462f" src="en-us_image_0000001295900168.png"></span></p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1121.html">Input Operators</a></div>
|
|
</div>
|
|
</div>
|
|
|