forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
125 lines
17 KiB
HTML
125 lines
17 KiB
HTML
<a name="mrs_01_1816"></a><a name="mrs_01_1816"></a>
|
|
|
|
<h1 class="topictitle1">Submitting Other Jobs</h1>
|
|
<div id="body1591347168766"><div class="section" id="mrs_01_1816__s1c1e36ce13e5426abc3cd3531ba390aa"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1816__afc6acfb0597c49ec9efb76228af00887">In addition to Hive, Spark2x, and Loader jobs, MapReduce, Java, Shell, HDFS, SSH, SubWorkflow, Streaming, and scheduled jobs can be submitted using the Oozie client.</p>
|
|
<div class="note" id="mrs_01_1816__nba5c808e1fb8407880609370781494e4"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1816__ace5fec53a0bc44bcb7c6e520e66c9585">You are advised to download the latest client.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1816__s4380d8b25bd04ff499a1360c982572f8"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1816__u6b695744b4a44d899e5372de89b099d0"><li id="mrs_01_1816__l36dd4642c4ab4215a7ab7571b9471184">The Oozie component and its client have been installed and are running properly.</li><li id="mrs_01_1816__l2edec5d9e8c4471abb7f9e8c9f03aadf">You have created or obtained the human-machine account and password for accessing the Oozie service.<div class="note" id="mrs_01_1816__note1131799592824"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1816__ul3475309792824"><li id="mrs_01_1816__li4434242292824">Shell job:<p id="mrs_01_1816__p6353748392824"><a name="mrs_01_1816__li4434242292824"></a><a name="li4434242292824"></a>This user must belong to the <strong id="mrs_01_1816__b167941223183016">hadoop</strong> and <strong id="mrs_01_1816__b38012238308">supergroup</strong> groups and be assigned the Oozie role operation permission. The Shell script must have the execution permission on each NodeManager.</p>
|
|
</li><li id="mrs_01_1816__li3496644292824">SSH job:<p id="mrs_01_1816__p4626252292824"><a name="mrs_01_1816__li3496644292824"></a><a name="li3496644292824"></a>This user must belong to the <strong id="mrs_01_1816__b197061362301">hadoop</strong> and <strong id="mrs_01_1816__b1471210365302">supergroup</strong> groups and be assigned the Oozie role operation permission. The mutual trust configuration is complete.</p>
|
|
</li><li id="mrs_01_1816__li1370951792824">Other jobs:<p id="mrs_01_1816__p5627679792824"><a name="mrs_01_1816__li1370951792824"></a><a name="li1370951792824"></a>This user must belong to the <strong id="mrs_01_1816__b23321344173017">hadoop</strong> and <strong id="mrs_01_1816__b1133894417306">supergroup</strong> groups and be assigned the Oozie role operation permission and other required permissions.</p>
|
|
</li><li id="mrs_01_1816__li536774108334">This user must also be assigned the <strong id="mrs_01_1816__b184677201317">manager_viewer</strong> role at least.</li></ul>
|
|
</div></div>
|
|
</li><li id="mrs_01_1816__ladc7b9ed3c3b46f6a1138daf800558ef">You have obtained the URL of the Oozie server (any instance) in the running state, for example, <strong id="mrs_01_1816__b067582420311">https://10.1.130.10:21003/oozie</strong>.</li><li id="mrs_01_1816__lc85e9ca5785d4cb2a27ef7ddb21e470b">You have obtained the name of the Oozie server, for example, <strong id="mrs_01_1816__b2724172913314">10-1-130-10</strong>.</li><li id="mrs_01_1816__lbf054015e3274171a4e5651db635ae9b">You have obtained the IP address of the active Yarn ResourceManager, for example, <strong id="mrs_01_1816__b127825326318">10.1.130.11</strong>.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1816__sd59ec33e524d4e09aead454d67568953"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1816__o1416603b0c3d483a92e810ef867e8268"><li id="mrs_01_1816__l350a1a2b7ba44961ba3fb6425d24bcd1"><span>Log in to the node where the Oozie client is installed as the client installation user.</span></li><li id="mrs_01_1816__ldb24e20d02fb49b8ab154c53a86ad2eb"><span>Run the following command to obtain the installation environment. <span class="filepath" id="mrs_01_1816__filepath9821934046"><b>/opt/<span id="mrs_01_1816__text2821334941">c</span>lient/</b></span> is an example client installation path.</span><p><p id="mrs_01_1816__aeb8aeb37c0ef4522b7e707624a46ffdb"><strong id="mrs_01_1816__ab3d3b59c3d4745739c53a8bdaa67b63e">source /opt/client/bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1816__l2af946bf1af44927ad288eed5f6517d3"><span>Check the cluster authentication mode.</span><p><ul id="mrs_01_1816__ude5a82d1f0094edfb94ecfd0d59d7818"><li id="mrs_01_1816__l7bb7f1e8ee944d1ea3e04df6848dde84">If the cluster is in security mode, run the <strong id="mrs_01_1816__b1690115243117">kinit</strong> command to authenticate users.<p id="mrs_01_1816__ad86dba79553f488099f4cc1ed5fd6192">For example, the <strong id="mrs_01_1816__b335975413110">oozieuser</strong> user is authenticated using the following command:</p>
|
|
<p id="mrs_01_1816__a22311f7f5d0747079f375239f6bf5b4c"><strong id="mrs_01_1816__b42354349103612">kinit oozieuser</strong></p>
|
|
</li><li id="mrs_01_1816__l34506fef00124b3d8fbf3802fb26b479">If the cluster is in normal mode, go to <a href="#mrs_01_1816__lc7fe1634fbdc45b592872850a4bd9def">4</a>.</li></ul>
|
|
</p></li><li id="mrs_01_1816__lc7fe1634fbdc45b592872850a4bd9def"><a name="mrs_01_1816__lc7fe1634fbdc45b592872850a4bd9def"></a><a name="lc7fe1634fbdc45b592872850a4bd9def"></a><span>Go to the example directory based on the type of the task you submit.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1816__t939267bfc02145b4a6e6bc523644d90c" frame="border" border="1" rules="all"><caption><b>Table 1 </b>List of example directories</caption><thead align="left"><tr id="mrs_01_1816__rd6489cca7b534bd98da9eb91546b939a"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.2.4.2.1.2.3.1.1"><p id="mrs_01_1816__a6000469e20d64a538ab7eb51dd5d0758">Job Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="75%" id="mcps1.3.3.2.4.2.1.2.3.1.2"><p id="mrs_01_1816__a64d3acf19a6d47ce90fd20c73772e4fd">Example Directory</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1816__r283ea753d4e24f1eb196cba66397b08e"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_1816__a51e98fb6c9a349e39201473ab60d1d7f">MapReduce job</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_1816__a4ab33c4439e946e38701af0c19e8693d"><i><span class="varname" id="mrs_01_1816__varname4442383613">Client installation directory</span></i><strong id="mrs_01_1816__b148874610410">/Oozie/oozie-client-*/examples/apps/map-reduce</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__rbf0f2c2869fb4dbc91c7b37591a54ee9"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_1816__a056265b73d3c46afb83a22b3d397448b">Java job</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_1816__ab03abb69db284c0da9d512fd272a8e9b"><i><span class="varname" id="mrs_01_1816__varname27781043163612">Client installation directory</span></i><strong id="mrs_01_1816__b845010910510">/Oozie/oozie-client-*/examples/apps/java-main</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__ra7079b30202045f49c22c539c43b485d"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_1816__ac4ff1a881b8a480aaca748414bd116a8">Shell job</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_1816__a326af45bf83d402996907709bdf1a243"><i><span class="varname" id="mrs_01_1816__varname1129611911377">Client installation directory</span></i><strong id="mrs_01_1816__b134081220357">/Oozie/oozie-client-*/examples/apps/shell</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__rb535b19d8a8e4bed93eca106d8046575"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_1816__aeb9b088f4aef4743b65949070e9af290">Streaming job</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_1816__aba5e3b74fb2f40ebbb1e35a92b94371f"><i><span class="varname" id="mrs_01_1816__varname97709339371">Client installation directory</span></i><strong id="mrs_01_1816__b206961928750">/Oozie/oozie-client-*/examples/apps/shell</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__rf659da179c2444f7b9be8ec834ff059e"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_1816__acba2a4da863948f79bb2d709b3f933b4">SubWorkflow job</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_1816__a697ae0794536421dbbf34eeac51a165f"><i><span class="varname" id="mrs_01_1816__varname20614353143713">Client installation directory</span></i><strong id="mrs_01_1816__b185151836650">/Oozie/oozie-client-*/examples/apps/subwf</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__rf660a1dc903744b2b17b7de6a4cb7dbf"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_1816__aa578ee9f83244cf5ae2ebe78734dc90b">SSH job</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_1816__a07163eaab1bd491cbf24493e6ce7a014"><i><span class="varname" id="mrs_01_1816__varname133421666383">Client installation directory</span></i><strong id="mrs_01_1816__b1209945856">/Oozie/oozie-client-*/examples/apps/ssh</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__r32e5b71a4c5e498c86750ed212d5e11d"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.4.2.1.2.3.1.1 "><p id="mrs_01_1816__a6bc006a9d307446395606699a409364a">Scheduled job</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="75%" headers="mcps1.3.3.2.4.2.1.2.3.1.2 "><p id="mrs_01_1816__aad8dd6dbe0784853bbbde3ce56df6e40"><i><span class="varname" id="mrs_01_1816__varname3550101923818">Client installation directory</span></i><strong id="mrs_01_1816__b621955313516">/Oozie/oozie-client-*/examples/apps/cron</strong></p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_1816__n2ae677a6a25d40f1beb27ea254fe3dcf"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1816__a6577fd0174634a1b97655f2ffddb162c">The examples of other jobs contain HDFS job examples.</p>
|
|
</div></div>
|
|
<p id="mrs_01_1816__af8b646021f5646cd82c7c85ab77f92fa"><a href="#mrs_01_1816__t4569c24069cf4e6b908ce637a4b6af6d">Table 2</a> lists the files that you need to pay attention to in the example directory.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1816__t4569c24069cf4e6b908ce637a4b6af6d"></a><a name="t4569c24069cf4e6b908ce637a4b6af6d"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1816__t4569c24069cf4e6b908ce637a4b6af6d" frame="border" border="1" rules="all"><caption><b>Table 2 </b>File description</caption><thead align="left"><tr id="mrs_01_1816__r40b98c367d904fd49c59d37f2d2e64f8"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.4.2.3.1.1"><p id="mrs_01_1816__a094957f26a464c50817a32510e0c36af">File</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.4.2.4.2.3.1.2"><p id="mrs_01_1816__a510b22ee17c042dbb2eb8c223cbd5f91">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1816__r9d259ffb7eb74530a3b8d58dc70c7c16"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.1 "><p id="mrs_01_1816__a9fe5512eefc44efc846148c98a6d8172">job.properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.2 "><p id="mrs_01_1816__ad10161ee2be74d6abbb21609e673ef29">Parameter definition file of a workflow</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__rba634f89678f4112afc60cf105a1b626"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.1 "><p id="mrs_01_1816__a2e51cf1b5a3f4b8bba6f6bc0b72bf0e9">workflow.xml</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.2 "><p id="mrs_01_1816__a673397c008bd489a8cdcd3a682accebd">Rule definition file of a workflow</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__r4605bcb7c2ee488f80e72ba5fcb7df6c"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.1 "><p id="mrs_01_1816__aba387759b8744c57970e447333f5d495">lib</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.2 "><p id="mrs_01_1816__ada5a34ac5693496cbe320ffe5d4bf8de">Directory of the JAR file on which a workflow depends</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__r5a7b06839e9a4d719aff1c8bd4ec4462"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.1 "><p id="mrs_01_1816__a212dd430d83c4ebda1db7774754d12db">coordinator.xml</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.2 "><p id="mrs_01_1816__a0d669d728b574ee28725ac35ce6cfd5c">Scheduled job configuration file which can be used to set a scheduled policy. The file is in the <span class="filepath" id="mrs_01_1816__filepath1818545173811"><b>cron</b></span> directory.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1816__r3657b7706d464deda6541f3158800c31"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.1 "><p id="mrs_01_1816__af1b370ff106445c285ee9f2c9eb744fc">oozie_shell.sh</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.4.2.4.2.3.1.2 "><p id="mrs_01_1816__a0f4dbd970ef947f789eb6fd6ecc053f3">Shell script file required for submitting shell jobs. The file is in the <span class="filepath" id="mrs_01_1816__filepath81806710397"><b>shell</b></span> directory.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_1816__l09e5b14029f148608c492348086924d2"><span>Run the following command to edit the <span class="filepath" id="mrs_01_1816__filepath63861013103917"><b>job.properties</b></span> file:</span><p><p id="mrs_01_1816__a3f7a6cd47cd9439cb0571e3d14aa7e08"><strong id="mrs_01_1816__a45530678b551475c82c5ea4255b05700">vi job.properties</strong></p>
|
|
<p id="mrs_01_1816__a9f71cb8dbb1e4b3cab603396a9fac28e">Perform the following modifications:</p>
|
|
<p id="mrs_01_1816__a68826410502a4db6bb371211ef5a93ba">Change the value of <span class="parmname" id="mrs_01_1816__parmname752951919395"><b>userName</b></span> to the name of the human-machine user who submits the job, for example, <span class="parmvalue" id="mrs_01_1816__parmvalue1853513194395"><b>userName=oozieuser</b></span>.</p>
|
|
</p></li><li id="mrs_01_1816__l7edd8677afbd42569ba6eaabdd104c8d"><span>Run the <strong id="mrs_01_1816__b13982122314398">oozie job</strong> command to run the workflow file:</span><p><p id="mrs_01_1816__aea32d17316e740c7a29d80c69fc7a96c"><strong id="mrs_01_1816__b26261225103918">oozie job -oozie https://</strong><em id="mrs_01_1816__i11627142510393">Host name of the oozie role</em><strong id="mrs_01_1816__b2062822515392">:21003/oozie -config </strong><i><span class="varname" id="mrs_01_1816__varname1262892518392">File path of job.properties</span></i><strong id="mrs_01_1816__b20629152593918"> -run</strong></p>
|
|
<p id="mrs_01_1816__aecf2d44835304c519940fd3ba3d29217">Example:</p>
|
|
<p id="mrs_01_1816__a8e581a443419419d905e40adef82edd8"><strong id="mrs_01_1816__ae6f3d34ddf474eec8ee715c51299d939">oozie job -oozie https://10-1-130-10:21003/oozie -config</strong></p>
|
|
<p id="mrs_01_1816__a4cacee7c96d146b7b4c1f57d26ce988c"><strong id="mrs_01_1816__a1eda3e0eaf7045f09b2151ca3cd22d70">/opt/client/Oozie/oozie-client-*/examples/apps/map-reduce/job.properties -run</strong></p>
|
|
<div class="note" id="mrs_01_1816__n026739d7fb0249169bfb053afb3f9f0f"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1816__u3e1c54ff1044497595dc2eb851641de5"><li id="mrs_01_1816__lc5e4cc6c22e44df2a67b79fb0517dddc">The command parameters are described as follows:<p id="mrs_01_1816__aa21f980ff7654a068c2e376cc3385292"><a name="mrs_01_1816__lc5e4cc6c22e44df2a67b79fb0517dddc"></a><a name="lc5e4cc6c22e44df2a67b79fb0517dddc"></a><strong id="mrs_01_1816__b89462035203918">-oozie</strong> URL of the Oozie server that executes a job</p>
|
|
<p id="mrs_01_1816__ae93001c22a4d47cc890a017ea385732c"><strong id="mrs_01_1816__b727833713394">-config</strong> Workflow property file</p>
|
|
<p id="mrs_01_1816__a6966a0b8916a403aa835ef24ef1483be"><strong id="mrs_01_1816__b12632154083911">-run</strong> Executing a workflow</p>
|
|
</li><li id="mrs_01_1816__ld5025afa2d17467a90a3ac09e1250c24">If a job ID, for example, <strong id="mrs_01_1816__b169791641153914">job: 0000021-140222101051722-oozie-omm-W</strong>, is displayed after the workflow file is executed, the job is successfully submitted. You can view the execution results on the Oozie management page.<p id="mrs_01_1816__aa256268feb4d41cf8449fcda84b3fba2">Log in to the Oozie web UI at <strong id="mrs_01_1816__b85722045123918">https</strong>://<em id="mrs_01_1816__i1557714510395">IP address of the Oozie role</em><strong id="mrs_01_1816__b17578145153919">:21003/oozie</strong> as user <strong id="mrs_01_1816__b205785450390">oozieuser</strong>.</p>
|
|
<p id="mrs_01_1816__aa372c709847d4c2bafdf5db07a9c52ce">On the Oozie web UI, you can view the submitted workflow information based on the job ID in the table on the page.</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1812.html">Using Oozie Client to Submit an Oozie Job</a></div>
|
|
</div>
|
|
</div>
|
|
|