forked from docs/doc-exports
Yang, Tong
48706b7552
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
63 lines
14 KiB
HTML
63 lines
14 KiB
HTML
<a name="mrs_01_1815"></a><a name="mrs_01_1815"></a>
|
|
|
|
<h1 class="topictitle1">Submitting a Loader Job with Oozie Client</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_1815__en-us_topic_0000001173949420_sf9f0669ef0594cd6827a6a1945fa39a9"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1815__en-us_topic_0000001173949420_a08ffd82c40204fc993699dc7ae64074f">This section describes how to submit a Loader job using the Oozie client.</p>
|
|
<div class="note" id="mrs_01_1815__en-us_topic_0000001173949420_n26baa769b8864badad9bbd55bf8b6be9"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1815__en-us_topic_0000001173949420_a05b297fa9fe84d8fad4eb4ccb71cceb5">You are advised to download the latest client.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1815__en-us_topic_0000001173949420_s2a4972530b7942d3884e8f2fd506720b"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1815__en-us_topic_0000001173949420_u2cd59a22b503403193bd3d459ca58582"><li id="mrs_01_1815__en-us_topic_0000001173949420_l5840702206d44a63a51b020bb5878697">The Hive and Oozie components and clients have been installed and are running properly.</li><li id="mrs_01_1815__en-us_topic_0000001173949420_l0e56a77e8e4949bea73a8807d9440792">You have created or obtained the human-machine account and password for accessing the Oozie service.<div class="note" id="mrs_01_1815__en-us_topic_0000001173949420_note315568028334"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1815__en-us_topic_0000001173949420_ul155757648334"><li id="mrs_01_1815__en-us_topic_0000001173949420_li59641568334">This user must belong to the <strong id="mrs_01_1815__en-us_topic_0000001173949420_b2651343172618">hadoop</strong>, <strong id="mrs_01_1815__en-us_topic_0000001173949420_b6711643162617">supergroup</strong>, and <strong id="mrs_01_1815__en-us_topic_0000001173949420_b271134314269">hive</strong> groups and be assigned with the Oozie role operation permission.</li><li id="mrs_01_1815__en-us_topic_0000001173949420_li536774108334">This user must also be assigned the <strong id="mrs_01_1815__en-us_topic_0000001173949420_b10823204662616">manager_viewer</strong> role at least.</li></ul>
|
|
</div></div>
|
|
</li><li id="mrs_01_1815__en-us_topic_0000001173949420_le943cf5dd33b43e2a6f966850543ace3">You have obtained the URL of the Oozie server (any instance) in the running state, for example, <strong id="mrs_01_1815__en-us_topic_0000001173949420_b8681748152612">https://10.1.130.10:21003/oozie</strong>.</li><li id="mrs_01_1815__en-us_topic_0000001173949420_l22ab7f9f32144d05a50b016af47e736f">You have obtained the name of the Oozie server, for example, <strong id="mrs_01_1815__en-us_topic_0000001173949420_b13425150202617">10-1-130-10</strong>.</li><li id="mrs_01_1815__en-us_topic_0000001173949420_lad6db5ec38cd4bbcb700e9b6a73ca8d5">You have obtained the IP address of the active Yarn ResourceManager, for example, <strong id="mrs_01_1815__en-us_topic_0000001173949420_b208881851152611">10.1.130.11</strong>.</li><li id="mrs_01_1815__en-us_topic_0000001173949420_l18579f5b7e0c4fd3a52ff594488af265">You have created a Loader job to be scheduled and obtained the job ID.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1815__en-us_topic_0000001173949420_s57ae1abcc818413d88ca1c2ae5e4a498"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1815__en-us_topic_0000001173949420_oa0e1ad1ca5ab4a83afb8b7e618984c20"><li id="mrs_01_1815__en-us_topic_0000001173949420_l09be4bd6fa7648d4a27e62aa97de13fc"><span>Log in to the node where the Oozie client is installed as the client installation user.</span></li><li id="mrs_01_1815__en-us_topic_0000001173949420_l6884fe8bf8994f48b3db8ffcaf746f88"><span>Run the following command to obtain the installation environment. In the command, <span class="filepath" id="mrs_01_1815__en-us_topic_0000001173949420_fa4671ca50bde483887d472647a69fc48"><b>/opt/<span id="mrs_01_1815__en-us_topic_0000001173949420_text1742104015911">c</span>lient/</b></span> indicates the client installation path.</span><p><p id="mrs_01_1815__en-us_topic_0000001173949420_a0f1b5d06f7874d8c9239c40161f7245b"><strong id="mrs_01_1815__en-us_topic_0000001173949420_ab1d979876452450ea8d987cbb0dab04d">source /opt/client/bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1815__en-us_topic_0000001173949420_l8700468e24024996a4149dc0a37eac8f"><span>Check the cluster authentication mode.</span><p><ul id="mrs_01_1815__en-us_topic_0000001173949420_u2bf63897c7a14c3886fc1a9702d01db4"><li id="mrs_01_1815__en-us_topic_0000001173949420_lded70913584c45d5bbbf9a2111c50c66">If the cluster is in security mode, run the <strong id="mrs_01_1815__en-us_topic_0000001173949420_b147451646172715">kinit</strong> command to authenticate users.<p id="mrs_01_1815__en-us_topic_0000001173949420_ac63c0a2720a842b59a4a9e6935b8aabf">For example, the <strong id="mrs_01_1815__en-us_topic_0000001173949420_b92011448102713">oozieuser</strong> user is authenticated using the following command:</p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_adafc8c7bf9614279b476239e9a6e1d77"><strong id="mrs_01_1815__en-us_topic_0000001173949420_b58933632103556">kinit oozieuser</strong></p>
|
|
</li><li id="mrs_01_1815__en-us_topic_0000001173949420_lfb671433802145ed9dbadddd8dd9ff62">If the cluster is in normal mode, go to <a href="#mrs_01_1815__en-us_topic_0000001173949420_l91970df22ff342c59cfacd6aa7a436d9">4</a>.</li></ul>
|
|
</p></li><li id="mrs_01_1815__en-us_topic_0000001173949420_l91970df22ff342c59cfacd6aa7a436d9"><a name="mrs_01_1815__en-us_topic_0000001173949420_l91970df22ff342c59cfacd6aa7a436d9"></a><a name="en-us_topic_0000001173949420_l91970df22ff342c59cfacd6aa7a436d9"></a><span>Run the following command to go to the example directory:</span><p><p id="mrs_01_1815__en-us_topic_0000001173949420_ae82d00e4d8684f0db9360c4b092fce7c"><strong id="mrs_01_1815__en-us_topic_0000001173949420_a5bd44a89e1894b34a19db5b11bc18cec">cd /opt/client/Oozie/oozie-client-*/examples/apps/sqoop/</strong></p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_a70fac9924e6b4fca9700586444fe6b9e"><a href="#mrs_01_1815__en-us_topic_0000001173949420_t088bd902005a48549ec48b463bbb9278">Table 1</a> lists the files that you need to pay attention to in the directory.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1815__en-us_topic_0000001173949420_t088bd902005a48549ec48b463bbb9278"></a><a name="en-us_topic_0000001173949420_t088bd902005a48549ec48b463bbb9278"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1815__en-us_topic_0000001173949420_t088bd902005a48549ec48b463bbb9278" frame="border" border="1" rules="all"><caption><b>Table 1 </b>File description</caption><thead align="left"><tr id="mrs_01_1815__en-us_topic_0000001173949420_rfa8e7a6c3b0640f180c2db60fdd1920a"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.3.2.3.1.1"><p id="mrs_01_1815__en-us_topic_0000001173949420_aeb6893b482ed4d0ba373c30516a41d53">File</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.3.2.3.1.2"><p id="mrs_01_1815__en-us_topic_0000001173949420_a1943b7073cfa4987861d1c84f54433d7">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1815__en-us_topic_0000001173949420_r65ee9aadf75c46419d92b20fa590cf1c"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.1 "><p id="mrs_01_1815__en-us_topic_0000001173949420_ac31f1a7cba1140ae81aae4d5c9895347">job.properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.2 "><p id="mrs_01_1815__en-us_topic_0000001173949420_ad60bfcbd20394dea84e7213e7df95aae">Parameter definition file of a workflow</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1815__en-us_topic_0000001173949420_rf2ed451342e243f5abf92c275c5756f2"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.1 "><p id="mrs_01_1815__en-us_topic_0000001173949420_adee19255dea74de7bdb28071238d6d6c">workflow.xml</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.3.2.3.1.2 "><p id="mrs_01_1815__en-us_topic_0000001173949420_a40d65130d6cd489eb3f9c5f52344ab00">Rule definition file of a workflow</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_1815__en-us_topic_0000001173949420_l946a7b39c1dd43cb97999c6a4a27b9c3"><span>Run the following command to edit the <span class="filepath" id="mrs_01_1815__en-us_topic_0000001173949420_filepath18068212812"><b>job.properties</b></span> file:</span><p><p id="mrs_01_1815__en-us_topic_0000001173949420_a74bf1e98265248aea488442321f87af1"><strong id="mrs_01_1815__en-us_topic_0000001173949420_ae6dd1f9bbf3f46e5a85be18cabd6feb4">vi job.properties</strong></p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_aceb51a67f59649a7869cae14fd7eed14">Perform the following modifications:</p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_a2789dca82fc8454282fa1682532d8d5e">Change the value of <span class="parmname" id="mrs_01_1815__en-us_topic_0000001173949420_parmname176781766281"><b>userName</b></span> to the name of the human-machine user who submits the job, for example, <span class="parmvalue" id="mrs_01_1815__en-us_topic_0000001173949420_parmvalue186833616285"><b>userName=oozieuser</b></span>.</p>
|
|
</p></li><li id="mrs_01_1815__en-us_topic_0000001173949420_la0c4783a49c74114a3e30cf0daaf48b8"><span>Run the following command to edit the <span class="filepath" id="mrs_01_1815__en-us_topic_0000001173949420_filepath1895614982815"><b>workflow.xml</b></span> file:</span><p><p id="mrs_01_1815__en-us_topic_0000001173949420_a4ee3d10d8dc347f083608772a8dea5c9"><strong id="mrs_01_1815__en-us_topic_0000001173949420_ada08fd41723d424e9722adec2f18a886">vi workflow.xml</strong></p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_a1cdca397ceb64663b77ffbefa44e285b">Perform the following modifications:</p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_a775c8a2a8c08448a9c86952df4af4140">Change the value of <span class="parmname" id="mrs_01_1815__en-us_topic_0000001173949420_parmname121341511281"><b>command</b></span> to the ID of the Loader job to be scheduled, for example, <strong id="mrs_01_1815__en-us_topic_0000001173949420_b191901511285">1</strong>.</p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_p12877195713449">Upload the <span class="filepath" id="mrs_01_1815__en-us_topic_0000001173949420_filepath19160161762816"><b>workflow.xml</b></span> file to the HDFS path in the <strong id="mrs_01_1815__en-us_topic_0000001173949420_b616131717287">job.properties</strong> file.</p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_p5195164518463"><strong id="mrs_01_1815__en-us_topic_0000001173949420_b3793153417255">hdfs dfs -put -f workflow.xml /user/userName/examples/apps/sqoop</strong></p>
|
|
</p></li><li id="mrs_01_1815__en-us_topic_0000001173949420_l1159eb6dca804bb3911fc8b4d72bfb4f"><span>Run the <strong id="mrs_01_1815__en-us_topic_0000001173949420_b13652920132819">oozie job</strong> command to run the workflow file:</span><p><p id="mrs_01_1815__en-us_topic_0000001173949420_p54529342154411"><strong id="mrs_01_1815__en-us_topic_0000001173949420_b5549221152818">oozie job -oozie https://</strong><em id="mrs_01_1815__en-us_topic_0000001173949420_i1255002172814">Host name of the Oozie role</em><strong id="mrs_01_1815__en-us_topic_0000001173949420_b1755011219281">:21003/oozie/ -config job.properties -run</strong></p>
|
|
<div class="note" id="mrs_01_1815__en-us_topic_0000001173949420_n5a47cb09ca3a450c83630135e7e94d6c"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1815__en-us_topic_0000001173949420_u6d4fc636f0704af8823958ca92189ab3"><li id="mrs_01_1815__en-us_topic_0000001173949420_la27e2a68012b44b8bab2b663cd1bcb24">The command parameters are described as follows:<p id="mrs_01_1815__en-us_topic_0000001173949420_add0178af9cb74a7c8bc43ccde17e8056"><a name="mrs_01_1815__en-us_topic_0000001173949420_la27e2a68012b44b8bab2b663cd1bcb24"></a><a name="en-us_topic_0000001173949420_la27e2a68012b44b8bab2b663cd1bcb24"></a><strong id="mrs_01_1815__en-us_topic_0000001173949420_b92301726172816">-oozie</strong> URL of the Oozie server that executes a job</p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_ac71440e8b53147bcb42b441a9d6f02cf"><strong id="mrs_01_1815__en-us_topic_0000001173949420_b1849919277289">-config</strong> Workflow property file</p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_a80ac946821e648c8b67f4d19e7a48dc0"><strong id="mrs_01_1815__en-us_topic_0000001173949420_b566129162813">-run</strong> Executing a workflow</p>
|
|
</li><li id="mrs_01_1815__en-us_topic_0000001173949420_l6ff25fccfc6048aaa65465b9e8d3485d">If a job ID, for example, <strong id="mrs_01_1815__en-us_topic_0000001173949420_b1064530152813">job: 0000021-140222101051722-oozie-omm-W</strong>, is displayed after the workflow file is executed, the job is successfully submitted. You can view the execution results on the Oozie management page.<p id="mrs_01_1815__en-us_topic_0000001173949420_a1f83cb1187e44f6c82e4776e5c163925">Log in to the Oozie web UI at <strong id="mrs_01_1815__en-us_topic_0000001173949420_b3728338282">https</strong>://<em id="mrs_01_1815__en-us_topic_0000001173949420_i177723312810">IP address of the Oozie role</em><strong id="mrs_01_1815__en-us_topic_0000001173949420_b197813318283">:21003/oozie</strong> as user <strong id="mrs_01_1815__en-us_topic_0000001173949420_b167813372815">oozieuser</strong>.</p>
|
|
<p id="mrs_01_1815__en-us_topic_0000001173949420_a9822d665c5c64c7bad8f238e6cbf0541">On the Oozie web UI, you can view the submitted workflow information based on the job ID in the table on the page.</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1812.html">Using Oozie Client to Submit an Oozie Job</a></div>
|
|
</div>
|
|
</div>
|
|
|