forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
100 lines
14 KiB
HTML
100 lines
14 KiB
HTML
<a name="mrs_01_1146"></a><a name="mrs_01_1146"></a>
|
|
|
|
<h1 class="topictitle1">Hive output</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_1146__en-us_topic_0000001173789680_se3ba2de14a25470b946d9d58c11c8d46"><h4 class="sectiontitle">Overview</h4><p id="mrs_01_1146__en-us_topic_0000001173789680_a4a16a085daf9414f96718bfda38e248b">The <span class="parmname" id="mrs_01_1146__en-us_topic_0000001173789680_pfe9adabd42414f0d94aa470d9c0312b2"><b>Hive Output</b></span> operator exports existing fields to specified columns of a Hive table.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1146__en-us_topic_0000001173789680_sc195946bc66b4b33a4099481b1d146c5"><h4 class="sectiontitle">Input and Output</h4><ul id="mrs_01_1146__en-us_topic_0000001173789680_u90f0c37c4c4d4b2e9f9bfbad628dce82"><li id="mrs_01_1146__en-us_topic_0000001173789680_l97a8a0e155f3447fb3b9a7e8edaec689">Input: fields to be exported</li><li id="mrs_01_1146__en-us_topic_0000001173789680_l061c591163e345c29953e2f85f4f3055">Output: Hive table</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1146__en-us_topic_0000001173789680_s0cc09958269948d18933ee929f96a58a"><h4 class="sectiontitle">Parameters</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1146__en-us_topic_0000001173789680_t24f73308a13a489f851c826b0029915a" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Operator parameters description</caption><thead align="left"><tr id="mrs_01_1146__en-us_topic_0000001173789680_racbdf175b3eb43d886c2171280a4aa42"><th align="left" class="cellrowborder" valign="top" width="18.37%" id="mcps1.3.3.2.2.6.1.1"><p id="mrs_01_1146__en-us_topic_0000001173789680_a1c713391a60f4a3b8ddbe584e4bc1842">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="54.730000000000004%" id="mcps1.3.3.2.2.6.1.2"><p id="mrs_01_1146__en-us_topic_0000001173789680_a27c229687b254b1bb0be21633dce1f60">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="7.5200000000000005%" id="mcps1.3.3.2.2.6.1.3"><p id="mrs_01_1146__en-us_topic_0000001173789680_a0274a28ca7a1443d9ecc81b43db809f5">Node Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="9.69%" id="mcps1.3.3.2.2.6.1.4"><p id="mrs_01_1146__en-us_topic_0000001173789680_a20b48a436ffe4d2d8407b29d0cfb145c">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="9.69%" id="mcps1.3.3.2.2.6.1.5"><p id="mrs_01_1146__en-us_topic_0000001173789680_aa0fc92ea1c9349d59cda62d69e6340dd">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1146__en-us_topic_0000001173789680_r4742ece39da34e8b8bf4c0a5e5796425"><td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.3.2.2.6.1.1 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a78fe894cbcc54d7d97fe18899203a6eb">Hive file storage format</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="54.730000000000004%" headers="mcps1.3.3.2.2.6.1.2 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a7400c971f4ec4c2197d49196fc49afe1">Hive configuration file storage format. CSV, ORC, and RC are supported at present.</p>
|
|
<div class="note" id="mrs_01_1146__en-us_topic_0000001173789680_ne3f39bb11c584024b176d8566e242fbc"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="mrs_01_1146__en-us_topic_0000001173789680_ul1790344311481"><li class="text" id="mrs_01_1146__en-us_topic_0000001173789680_li190354364819">Parquet is a column-based storage format. In this format, the output field names of Loader be the same as the field names in Hive tables.</li><li id="mrs_01_1146__en-us_topic_0000001173789680_li11308155014484">For Hive of versions later than 1.2.0, a field name, instead of field number, is used to parse ORC files. Therefore, the output field names of Loader must be the same as those in Hive tables.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.2.2.6.1.3 "><p id="mrs_01_1146__en-us_topic_0000001173789680_af64e9d643dfd4b3a9d3bba537aba2922">enum</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.4 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a2b74e50d2f9b42af90f1c5f3dc377143">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.5 "><p id="mrs_01_1146__en-us_topic_0000001173789680_aab4e5d71108442128addc3e71e2a4e8c">CSV</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1146__en-us_topic_0000001173789680_r6318450d2b6a45c98f547ce77920f422"><td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.3.2.2.6.1.1 "><p id="mrs_01_1146__en-us_topic_0000001173789680_ae9de5d5ae83c471f84dc522592adad79">Hive file compression format</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="54.730000000000004%" headers="mcps1.3.3.2.2.6.1.2 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a35e8ede9b7a44c4f9f7f6bcd5b5fdcd8">Hive table file compression format. Select a format from the drop-down list. If you select <span class="parmvalue" id="mrs_01_1146__en-us_topic_0000001173789680_p18569ba6418643049bef13da23f6eed6"><b>NONE</b></span> or do not set this parameter, data is not compressed.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.2.2.6.1.3 "><p id="mrs_01_1146__en-us_topic_0000001173789680_adff09863ce3644468a12a24f4a6444fb">enum</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.4 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a861e29d01ad4452f9ace2d468506090b">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.5 "><p id="mrs_01_1146__en-us_topic_0000001173789680_ad56d4fdeba0b4e3f97d1cbf8b3a1b0d5">NONE</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1146__en-us_topic_0000001173789680_rc86c6aca480f47768ffd9f4888c0c6a8"><td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.3.2.2.6.1.1 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a701f0a700f8743bfbce6d24dfbacf5e0">Hive ORC file version</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="54.730000000000004%" headers="mcps1.3.3.2.2.6.1.2 "><p id="mrs_01_1146__en-us_topic_0000001173789680_ae98d4461021047e89253e4dcb87967ec">Version of the ORC file (when the storage format of the Hive table file is ORC).</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.2.2.6.1.3 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a18c94ac1caa14ebbbf45ce8dc57cf301">enum</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.4 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a16a6a40b03c949b3adec601da8ba5d74">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.5 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a58091aa22a5e4335bbb6eb8cb6fff60b">0.12</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1146__en-us_topic_0000001173789680_ra43439df11394cb687ddccef90d349be"><td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.3.2.2.6.1.1 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a75fcbca079fb43688b7474ea3acd8da6">Output delimiter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="54.730000000000004%" headers="mcps1.3.3.2.2.6.1.2 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a288fe1d651474ed0800433db1ec86d48">Delimiter.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.2.2.6.1.3 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a0b71f4f5b56f41ae8941bd24788e0d71">string</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.4 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a2f69b6b5921a41c3a7485664339a1f5b">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.5 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a62c73ea2ae034ad6af0753f04d6c7e25">None.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1146__en-us_topic_0000001173789680_r530b9706d3b44751bef050b1a1fa2f2e"><td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.3.2.2.6.1.1 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a30bd68e4176941d2adb4efb6e6ed4317">Output fields</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="54.730000000000004%" headers="mcps1.3.3.2.2.6.1.2 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a1feadbc5a1624311a825e259c1afa2b6">Information about output fields:</p>
|
|
<ul id="mrs_01_1146__en-us_topic_0000001173789680_u2f5d3fa879b94dcda1eab44eb6d1a0c4"><li id="mrs_01_1146__en-us_topic_0000001173789680_l6a1d6efc7b924b8bb364fc7c58308bbd">position: Position of output fields.</li><li id="mrs_01_1146__en-us_topic_0000001173789680_l22c852e9bfa3464fa8f038619c299b57">field name: Names of output fields.</li><li id="mrs_01_1146__en-us_topic_0000001173789680_la1fb4e540b5a45549f86f59bfa85bfb0">type: Field type. If type is set to <span class="parmvalue" id="mrs_01_1146__en-us_topic_0000001173789680_parmvalue160865746691851"><b>DATE</b></span>, <span class="parmvalue" id="mrs_01_1146__en-us_topic_0000001173789680_parmvalue178300465091851"><b>TIME</b></span>, or <span class="parmvalue" id="mrs_01_1146__en-us_topic_0000001173789680_parmvalue3685052391851"><b>TIMESTAMP</b></span>, you must specify a time format. If type is set to other values, the time format is invalid. An example time format is <span class="filepath" id="mrs_01_1146__en-us_topic_0000001173789680_f4ee7074a68b4409f8410765904b03bd4"><b>yyyyMMdd HH:mm:ss</b></span>.</li><li id="mrs_01_1146__en-us_topic_0000001173789680_l24093c043ee5473cacfccd7e0cd9d222">decimal format: scale and precision of the decimal.</li><li id="mrs_01_1146__en-us_topic_0000001173789680_le47fd2bcea1e44b49e819e8117f35003">length: Field value length. If the actual field value is excessively long, the value is cut based on the configured length. When <span class="parmname" id="mrs_01_1146__en-us_topic_0000001173789680_parmname12891171963819"><b>type</b></span> is set to <span class="parmvalue" id="mrs_01_1146__en-us_topic_0000001173789680_parmvalue78911197389"><b>CHAR</b></span>, spaces are added to the field value for supplement if the actual field value length is less than the configured length. When <span class="parmname" id="mrs_01_1146__en-us_topic_0000001173789680_parmname389111912384"><b>type</b></span> is set to <span class="parmvalue" id="mrs_01_1146__en-us_topic_0000001173789680_parmvalue889121912384"><b>VARCHAR</b></span>, no space is added to the field value for supplement if the actual field value length is less than the configured length.</li><li id="mrs_01_1146__en-us_topic_0000001173789680_l52badcfd69a445e0867f152291550967">partition key: indicates whether a column is a partition column. You can specify zero or multiple partition columns. If multiple primary keys are configured, they are combined according to the configuration sequence.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.5200000000000005%" headers="mcps1.3.3.2.2.6.1.3 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a1bf4677ac8234e6f9bafab950b6e707f">map</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.4 "><p id="mrs_01_1146__en-us_topic_0000001173789680_a2e3699bc5b0a412bbe5c2bd58690f8ff">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.69%" headers="mcps1.3.3.2.2.6.1.5 "><p id="mrs_01_1146__en-us_topic_0000001173789680_ab1cb3d2a1bce4d0380d63ce3799f6a73">None</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1146__en-us_topic_0000001173789680_s6a96833c3df1420393d61d3be8bc61c1"><h4 class="sectiontitle">Data Processing Rule</h4><ul id="mrs_01_1146__en-us_topic_0000001173789680_u2cc10a3ce4af4b7ca11e9013bd0c0456"><li id="mrs_01_1146__en-us_topic_0000001173789680_lb6f85565bc51405a96c0562102b0e82a">The field values are exported to the Hive table.</li><li id="mrs_01_1146__en-us_topic_0000001173789680_l05f8ed09ac45429bbed344a5e5dee42d">If one or more columns are specified as partition columns, the <span class="parmname" id="mrs_01_1146__en-us_topic_0000001173789680_parmname40029423491851"><b>Partition Handlers</b></span> feature is displayed on the <span class="parmname" id="mrs_01_1146__en-us_topic_0000001173789680_parmname33716573791851"><b>To</b></span> page in Step 4 of the job configuration. <span class="parmvalue" id="mrs_01_1146__en-us_topic_0000001173789680_parmvalue159013828791851"><b>Partition Handlers</b></span> specifies the number of handlers for processing data partitioning.</li><li id="mrs_01_1146__en-us_topic_0000001173789680_l054e3f57fc7549b6ad034b501410d03f">If no column is designated as partition columns, input data does not need to be partitioned, and <span class="parmvalue" id="mrs_01_1146__en-us_topic_0000001173789680_p4dfa90a083364abf88f3341b36c260db"><b>Partition Handlers</b></span> is hidden by default.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1146__en-us_topic_0000001173789680_s6bdd3f42f38445c595cc6e6c2fb597b7"><h4 class="sectiontitle">Example</h4><p id="mrs_01_1146__en-us_topic_0000001173789680_af4abe7b515a445bf867ed191a39be1c6">Use the <span class="parmname" id="mrs_01_1146__en-us_topic_0000001173789680_p84a55af78ec44a3199e5a948bee4f8e7"><b>CSV File Input</b></span> operator to generate two fields A and B.</p>
|
|
<p id="mrs_01_1146__en-us_topic_0000001173789680_a01ef59f804904975a55aed22a4d4bc0a">The following figure shows the source file.</p>
|
|
<p id="mrs_01_1146__en-us_topic_0000001173789680_a473bc6ca18614999ba91d2e77ae57104"><span><img id="mrs_01_1146__en-us_topic_0000001173789680_ib9cf8c7ad17e4bd08843f8a6bc35ba12" src="en-us_image_0000001348739729.jpg"></span></p>
|
|
<p id="mrs_01_1146__en-us_topic_0000001173789680_a10a00df349a341d59a9da9cfd5d4102d">Configure the <span class="parmname" id="mrs_01_1146__en-us_topic_0000001173789680_parmname187905106691851"><b>Hive Output</b></span> operator to export a_str and b_str to the Hive table.</p>
|
|
<p id="mrs_01_1146__en-us_topic_0000001173789680_a6b94bce5fee44c21b9b894c77b98b78a"><span><img id="mrs_01_1146__en-us_topic_0000001173789680_image26723514272" src="en-us_image_0000001349139421.png"></span></p>
|
|
<p id="mrs_01_1146__en-us_topic_0000001173789680_a7b0eb4c080364ebb8062397076524644">After the execution is complete, view the table data.</p>
|
|
<p id="mrs_01_1146__en-us_topic_0000001173789680_a25e5999fd8604c739c6a80510c912021"><span><img id="mrs_01_1146__image344919554488" src="en-us_image_0000001388325592.png"></span></p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1145.html">Output Operators</a></div>
|
|
</div>
|
|
</div>
|
|
|