forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
74 lines
12 KiB
HTML
74 lines
12 KiB
HTML
<a name="mrs_01_1813"></a><a name="mrs_01_1813"></a>
|
|
|
|
<h1 class="topictitle1">Submitting a Hive Job</h1>
|
|
<div id="body1591347168562"><div class="section" id="mrs_01_1813__sa79fbfc891004a179072198992e6e5a1"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1813__a89611c31dbc04c9d9f558f0a3e21c009">This section describes how to use the Oozie client to submit a Hive job.</p>
|
|
<p id="mrs_01_1813__a0eb57d8bbc414052a1433142567f80dc">Hive jobs are divided into the following types:</p>
|
|
<ul id="mrs_01_1813__u8b40331b39f54db4bfb54c53d62bfdd2"><li id="mrs_01_1813__l7e6205b63ae041699826204f79d08b23">Hive job<p id="mrs_01_1813__a4bd74c4fcd93412db1acbda899c6f729"><a name="mrs_01_1813__l7e6205b63ae041699826204f79d08b23"></a><a name="l7e6205b63ae041699826204f79d08b23"></a>Hive job that is connected in JDBC mode</p>
|
|
</li><li id="mrs_01_1813__l5f152bccff9d4e01ba9fa00c484eb4a0">Hive2 job<p id="mrs_01_1813__a798abcf307a841cab32c08415ee52e02"><a name="mrs_01_1813__l5f152bccff9d4e01ba9fa00c484eb4a0"></a><a name="l5f152bccff9d4e01ba9fa00c484eb4a0"></a>Hive job that is connected in Beeline mode</p>
|
|
</li></ul>
|
|
<p id="mrs_01_1813__af811c9e48f3245ddb87a15c19bef4cf9">This section describes how to submit a Hive job using the Oozie client.</p>
|
|
<div class="note" id="mrs_01_1813__n05d79ebe028b4a00b426ebe400c8a486"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1813__ue61288aa93444798a0e53c49a0152b6d"><li id="mrs_01_1813__lcd52fd0589dd4d2688d7899fd5321435">The procedure for submitting a Hive2 job using the Oozie client is the same as that for submitting a Hive job. You only need to change <span class="parmname" id="mrs_01_1813__parmname14940848121815"><b>/Hive</b></span> in the procedure to <span class="parmname" id="mrs_01_1813__parmname89451148131815"><b>/Hive2</b></span>.<p id="mrs_01_1813__a57189cae88564c34ae3c36cf459dd32a">For example, if the Hive job running directory is <span class="filepath" id="mrs_01_1813__f79db6bcb85084ddc88da53cd3ac1d896"><b>/opt/<span id="mrs_01_1813__text231611534315">client</span>/Oozie/oozie-client-*/examples/apps/hive/</b></span>, then the running directory of Hive2 is <span class="filepath" id="mrs_01_1813__fc622fed08a874a479c54b717129177f6"><b>/opt/<span id="mrs_01_1813__text484012396517">c</span>lient/Oozie/oozie-client-*/examples/apps/hive2/</b></span>.</p>
|
|
</li><li id="mrs_01_1813__l993078ca7ebb4bf291224f63d8569426">You are advised to download the latest client.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1813__sb52bb05ffef14e688e3d95328da1e71a"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1813__u518439d6edd24317a88b2e4eca6bd24a"><li id="mrs_01_1813__led6c45c3a68f4b2fb6ed03edd765f7c5">The Hive and Oozie components and clients have been installed and are running properly.</li><li id="mrs_01_1813__l0f3202130e4f4126b47913447d80d1b9">You have created or obtained the human-machine account and password for accessing the Oozie service.<div class="note" id="mrs_01_1813__note315568028334"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1813__ul155757648334"><li id="mrs_01_1813__li59641568334">This user must belong to the <strong id="mrs_01_1813__b85271739121916">hadoop</strong>, <strong id="mrs_01_1813__b1753293941912">supergroup</strong>, and <strong id="mrs_01_1813__b1653312391196">hive</strong> groups and be assigned with the Oozie role operation permission. If the multi-instance function is enabled for Hive, the user must belong to a specific Hive instance group, for example, <strong id="mrs_01_1813__b2910134615191">hive3</strong>.</li><li id="mrs_01_1813__li536774108334">This user must also be assigned the <strong id="mrs_01_1813__b19576105281918">manager_viewer</strong> role at least.</li></ul>
|
|
</div></div>
|
|
</li><li id="mrs_01_1813__l67e29a0db3d64a01ba3c8f8a242fffe2">You have obtained the URL of the Oozie server (any instance) in the running state, for example, <strong id="mrs_01_1813__b21555851918">https://10.1.130.10:21003/oozie</strong>.</li><li id="mrs_01_1813__l99b45eddd6bd47d3bb30ccde42dd5c80">You have obtained the name of the Oozie server, for example, <strong id="mrs_01_1813__b107523622014">10-1-130-10</strong>.</li><li id="mrs_01_1813__l33cbb5dabf1d4b61ac772fc3ec3c3e4a">You have obtained the IP address of the active Yarn ResourceManager, for example, <strong id="mrs_01_1813__b5937152732019">10.1.130.11</strong>.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1813__s36e7c96ddcf84fce9dc8e4fe9a59b187"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1813__o9c583a464a8241e4a3c14c5999fa0dbb"><li id="mrs_01_1813__lf6da6ca4845648b0bdb6aba37ae12697"><span>Log in to the node where the Oozie client is installed as the client installation user.</span></li><li id="mrs_01_1813__l55e93b01787342e58cabb1fa5bb9ec26"><span>Run the following command to obtain the installation environment. <span class="filepath" id="mrs_01_1813__f1d6036b32a974054bd3ccaf4f2b6cf56"><b>/opt/<span id="mrs_01_1813__text12646155711512">c</span>lient/</b></span> is an example client installation path.</span><p><p id="mrs_01_1813__af5b53099d5fd4014acfa3f00e9cab8e8"><strong id="mrs_01_1813__a99007b20226a48b4afedc8e77fcdfcd9">source /opt/client/bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1813__l4e2ff2ca52d14deda1a13b7710c8629b"><span>Check the cluster authentication mode.</span><p><ul id="mrs_01_1813__u3e34c12bdff2410794dcedbbc50e9fc9"><li id="mrs_01_1813__ld3d266cadddb40d5a4744a5ef13dd183">If the cluster is in security mode, run the <strong id="mrs_01_1813__b1169913672111">kinit</strong> command to authenticate users.<p id="mrs_01_1813__a0f33fbb3e4614cf2b612db9c74ed258f">For example, the <strong id="mrs_01_1813__b141147101216">oozieuser</strong> user is authenticated using the following command:</p>
|
|
<p id="mrs_01_1813__abd05f9679672432489e3e30bac99aaec"><strong id="mrs_01_1813__b1491705310358">kinit oozieuser</strong></p>
|
|
</li><li id="mrs_01_1813__l4d82835ce8874479a683cab82b3d8b2f">If the cluster is in normal mode, go to <a href="#mrs_01_1813__l3776b8e1e65745afb4922494b3c8d467">4</a>.</li></ul>
|
|
</p></li><li id="mrs_01_1813__l3776b8e1e65745afb4922494b3c8d467"><a name="mrs_01_1813__l3776b8e1e65745afb4922494b3c8d467"></a><a name="l3776b8e1e65745afb4922494b3c8d467"></a><span>Run the following command to go to the example directory:</span><p><p id="mrs_01_1813__aa461b17e86f5451188da24c0055b467b"><strong id="mrs_01_1813__b450182254014">cd /opt/client/Oozie/oozie-client-</strong>*<strong id="mrs_01_1813__b85072218402">/examples/apps/hive/</strong></p>
|
|
<p id="mrs_01_1813__a5c14e5fb56c042f7a42ce16931b93d4a"><a href="#mrs_01_1813__t4392ed6fbf084b2ebdccfa4455b75503">Table 1</a> lists the files that you need to pay attention to in the directory.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1813__t4392ed6fbf084b2ebdccfa4455b75503"></a><a name="t4392ed6fbf084b2ebdccfa4455b75503"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1813__t4392ed6fbf084b2ebdccfa4455b75503" frame="border" border="1" rules="all"><caption><b>Table 1 </b>File description</caption><thead align="left"><tr id="mrs_01_1813__r7f2f35e645954db789976070a82b9542"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.3.2.3.1.1"><p id="mrs_01_1813__a5dc6d920e0c741f3aa3b372a76691ef4">File</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.3.2.3.1.2"><p id="mrs_01_1813__a814399561518498ab492dcfc7aa720e8">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1813__r92482a29b41841d69bed1525327cb5b1"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.1 "><p id="mrs_01_1813__add7cffefe94f42168d4e7f1bcf15993c">hive-site.xml</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.2 "><p id="mrs_01_1813__a7ccd5a96889f4c73986abb9304b2bca2">Configuration file of a Hive job</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1813__rd76fec8ec015432b884c167db1a32c86"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.1 "><p id="mrs_01_1813__a35665aba7fed4fc9998b996e944bddf7">job.properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.2 "><p id="mrs_01_1813__a313dfce450f84037ac9691d3c4d7fb7e">Parameter definition file of a workflow</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1813__re0994b3e26504cecbaa2e6740ae76f2f"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.1 "><p id="mrs_01_1813__a3cb92531c1884d3ea69f32292dba54f3">script.q</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.2 "><p id="mrs_01_1813__a10448fc139df4463829f0935a4da54d5">SQL script of a Hive job</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1813__r2106d015973e4da3bb2428c400be731f"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.1 "><p id="mrs_01_1813__aa9ca45b824304cbd8c1a10e6f57c991c">workflow.xml</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.2 "><p id="mrs_01_1813__aaed0229985894fccbadd637b3fbab6dc">Rule definition file of a workflow</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_1813__l18a5e0b49401437db9ba7c752d2c8b47"><span>Run the following command to edit the <span class="filepath" id="mrs_01_1813__filepath432659172111"><b>job.properties</b></span> file:</span><p><p id="mrs_01_1813__a23e751d5b384459ba367912de55b147b"><strong id="mrs_01_1813__a4e9bffaabe8646f386e5073a1e12a5ed">vi job.properties</strong></p>
|
|
<p id="mrs_01_1813__aa1bf7cbd6cc0415cafdd369d6e2adead">Perform the following modifications:</p>
|
|
<p id="mrs_01_1813__a6d1ac6aec8ca423c922077bb32632787">Change the value of <span class="parmname" id="mrs_01_1813__parmname396458192213"><b>userName</b></span> to the name of the human-machine user who submits the job, for example, <span class="parmvalue" id="mrs_01_1813__parmvalue297018818227"><b>userName=oozieuser</b></span>.</p>
|
|
</p></li><li id="mrs_01_1813__l8fcb5a5e16d04791a32236b7ecb23655"><span>Run the <strong id="mrs_01_1813__b07701308232">oozie job</strong> command to run the workflow file:</span><p><p id="mrs_01_1813__p54529342154411"><strong id="mrs_01_1813__b466213102313">oozie job -oozie https://</strong><em id="mrs_01_1813__i176677312235">Host name of the Oozie role</em><strong id="mrs_01_1813__b19668173122312">:21003/oozie/ -config job.properties -run</strong></p>
|
|
<div class="note" id="mrs_01_1813__n5cda98303d794a588475ad29cda3d7f4"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1813__udb2d566703084af08560cb3b6b4113b9"><li id="mrs_01_1813__lf9fe407748134a4da8943c0d7134eaae">The command parameters are described as follows:<p id="mrs_01_1813__a53e54d88bb5b4bfc8fb3a25206fc1e29"><a name="mrs_01_1813__lf9fe407748134a4da8943c0d7134eaae"></a><a name="lf9fe407748134a4da8943c0d7134eaae"></a><strong id="mrs_01_1813__b637181019237">-oozie</strong> URL of the Oozie server that executes a job</p>
|
|
<p id="mrs_01_1813__a29c29d4de75c4ef3a36d6910cbc7ef5e"><strong id="mrs_01_1813__b737213237238">-config</strong> Workflow property file</p>
|
|
<p id="mrs_01_1813__a65431621b80b4852a76eca834b0b4e09"><strong id="mrs_01_1813__b17204026132310">-run</strong> Executing a workflow</p>
|
|
</li><li id="mrs_01_1813__lc32ce00634484f1eb82cc3cc2e469ccc">If a job ID, for example, <strong id="mrs_01_1813__b13789828152319">job: 0000021-140222101051722-oozie-omm-W</strong>, is displayed after the workflow file is executed, the job is successfully submitted. You can view the execution results on the Oozie management page.<p id="mrs_01_1813__ab5441363567e41959e26559c9e8e1c0a">Log in to the Oozie web UI at <strong id="mrs_01_1813__b42231543122316">https</strong>://<em id="mrs_01_1813__i322974362310">IP address of the Oozie role</em><strong id="mrs_01_1813__b10230043112313">:21003/oozie</strong> as user <strong id="mrs_01_1813__b223014392312">oozieuser</strong>.</p>
|
|
<p id="mrs_01_1813__a0534c3cd42c94f1dbecb97a4fef54e82">On the Oozie web UI, you can view the submitted workflow information based on the job ID in the table on the page.</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1812.html">Using Oozie Client to Submit an Oozie Job</a></div>
|
|
</div>
|
|
</div>
|
|
|