forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
66 lines
10 KiB
HTML
66 lines
10 KiB
HTML
<a name="mrs_01_1814"></a><a name="mrs_01_1814"></a>
|
|
|
|
<h1 class="topictitle1">Submitting a Spark2x Job</h1>
|
|
<div id="body1591347168652"><div class="section" id="mrs_01_1814__sc367440a9bee4ae9a15baede2902cc54"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1814__ae05864fc43454b69888e41725935e74b">This section describes how to submit a Spark2x job using the Oozie client.</p>
|
|
<div class="note" id="mrs_01_1814__n55f8e36fffd341878eb6390e2a633e8f"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1814__a5d75fccba647482fb315a28ee027d87a">You are advised to download the latest client.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1814__s03008bf64de9424c83937093f6557919"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1814__u6075e1bead804d118aeb83a52bc084ca"><li id="mrs_01_1814__l59e2fa74571d485bb4861d104ade63ea">The Spark2x and Oozie components and clients have been installed and are running properly.<p id="mrs_01_1814__a809de3689b214dd0bdc79d7e35fbae9c"><a name="mrs_01_1814__l59e2fa74571d485bb4861d104ade63ea"></a><a name="l59e2fa74571d485bb4861d104ade63ea"></a>If the current client is an earlier version, you need to download and install the client again.</p>
|
|
</li><li id="mrs_01_1814__l8ff319393ec949d9ae45feea7e19d290">You have created or obtained the human-machine account and password for accessing the Oozie service.<div class="note" id="mrs_01_1814__note315568028334"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1814__ul155757648334"><li id="mrs_01_1814__li59641568334">This user must belong to the <strong id="mrs_01_1814__b1427217510243">hadoop</strong>, <strong id="mrs_01_1814__b6278135118247">supergroup</strong>, and <strong id="mrs_01_1814__b122781551132411">hive</strong> groups and be assigned with the Oozie role operation permission. If the multi-instance function is enabled for Hive, the user must belong to a specific Hive instance group, for example, <strong id="mrs_01_1814__b134031359182410">hive3</strong>.</li><li id="mrs_01_1814__li536774108334">This user must also be assigned the <strong id="mrs_01_1814__b141251942257">manager_viewer</strong> role at least.</li></ul>
|
|
</div></div>
|
|
</li></ul>
|
|
</div>
|
|
<ul id="mrs_01_1814__ul589917804210"><li id="mrs_01_1814__l67e29a0db3d64a01ba3c8f8a242fffe2">You have obtained the URL of the Oozie server (any instance) in the running state, for example, <strong id="mrs_01_1814__b79741712516">https://10.1.130.10:21003/oozie</strong>.</li><li id="mrs_01_1814__l99b45eddd6bd47d3bb30ccde42dd5c80">You have obtained the name of the Oozie server, for example, <strong id="mrs_01_1814__b10537161213256">10-1-130-10</strong>.</li><li id="mrs_01_1814__l33cbb5dabf1d4b61ac772fc3ec3c3e4a">You have obtained the IP address of the active Yarn ResourceManager, for example, <strong id="mrs_01_1814__b1933617156253">10.1.130.11</strong>.</li></ul>
|
|
<div class="section" id="mrs_01_1814__section737612414426"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1814__oc881cfc28d414306af1adbbbf4a00b1b"><li id="mrs_01_1814__l7935c910b03545e0a5867ca4d3b2c006"><span>Log in to the node where the Oozie client is installed as the client installation user.</span></li><li id="mrs_01_1814__lfce2496a67db4306afc7110e3058abea"><span>Run the following command to obtain the installation environment. <span class="filepath" id="mrs_01_1814__f1995adbdd35741e6a3a9f62f7fb7a90b"><b>/opt/client/</b></span> is an example client installation path.</span><p><p id="mrs_01_1814__a9fe0ea3f05954692816d1ed270ff6e25"><strong id="mrs_01_1814__ad8409f5d475a4974b1a119c7e03644bc">source /opt/client/bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1814__l1875b9398af340e1844211cfd2f8272d"><span>Check the cluster authentication mode.</span><p><ul id="mrs_01_1814__u553e435dd8f84be8a18ff92d6d24725a"><li id="mrs_01_1814__ld507b00926e44383b9343a4bc2adbeb3">If the cluster is in security mode, run the <strong id="mrs_01_1814__b152721325257">kinit</strong> command to authenticate users.<p id="mrs_01_1814__a6f9517958c83424ea524bf7781acc911">For example, the <strong id="mrs_01_1814__b1986318333252">oozieuser</strong> user is authenticated using the following command:</p>
|
|
<p id="mrs_01_1814__ae4d53a149513466187a4bf7a3651dda3"><strong id="mrs_01_1814__b218349103537">kinit oozieuser</strong></p>
|
|
</li><li id="mrs_01_1814__l4c6e11eac05d4977a9b756054fd46659">If the cluster is in normal mode, go to <a href="#mrs_01_1814__lcc62479277a945d99a305c3a8402a40d">4</a>.</li></ul>
|
|
</p></li><li id="mrs_01_1814__lcc62479277a945d99a305c3a8402a40d"><a name="mrs_01_1814__lcc62479277a945d99a305c3a8402a40d"></a><a name="lcc62479277a945d99a305c3a8402a40d"></a><span>Run the following command to go to the example directory:</span><p><p id="mrs_01_1814__a6a361a015c424467b72a12dcbb292aaf"><strong id="mrs_01_1814__a2ec98494ecb742d3aa186b2d18928c32">cd /opt/client/Oozie/oozie-client-*/examples/apps/spark2x/</strong></p>
|
|
<p id="mrs_01_1814__aa736f6a3b96b40c0b60acd1d8626c64f"><a href="#mrs_01_1814__ta1113e97d79c4106a91f5e20da6899e0">Table 1</a> lists the files that you need to pay attention to in the directory.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1814__ta1113e97d79c4106a91f5e20da6899e0"></a><a name="ta1113e97d79c4106a91f5e20da6899e0"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1814__ta1113e97d79c4106a91f5e20da6899e0" frame="border" border="1" rules="all"><caption><b>Table 1 </b>File description</caption><thead align="left"><tr id="mrs_01_1814__r4aa1a6ae8d4346328f92540884cca876"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.4.2.4.2.3.2.3.1.1"><p id="mrs_01_1814__a16a9e33c6cbd4b898c5205ffd33d6036">File</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.4.2.4.2.3.2.3.1.2"><p id="mrs_01_1814__ab8b5f25780904c488a54fa26e79a8446">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1814__rdb5a7205478d49d7a5ff58bb41798174"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.4.2.3.2.3.1.1 "><p id="mrs_01_1814__afb57a440373e48b8b06b50fa052385c7">job.properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.4.2.3.2.3.1.2 "><p id="mrs_01_1814__aa99f6c5fc08a4b558f4f5b658f73af02">Parameter definition file of a workflow</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1814__r280e2c0ce768422d970506fb7530754e"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.4.2.3.2.3.1.1 "><p id="mrs_01_1814__a11a9d0dfd7c34ae2bbef3a5513508fa0">workflow.xml</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.4.2.3.2.3.1.2 "><p id="mrs_01_1814__a6b5bf5dcae684e0c878e0c5c4afa0832">Rule definition file of a workflow</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1814__rdbc872b81a5042f1aef9b58013f286ff"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.4.2.3.2.3.1.1 "><p id="mrs_01_1814__a7605b6504ca144279277c82abe906f31">lib</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.4.2.4.2.3.2.3.1.2 "><p id="mrs_01_1814__a27f23152fa1244429fc40a22cee450d1">Directory of the JAR file on which a workflow depends</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_1814__lc81b755786704225844019d5c86d4966"><span>Run the following command to edit the <span class="filepath" id="mrs_01_1814__filepath180475414251"><b>job.properties</b></span> file:</span><p><p id="mrs_01_1814__aeb6f21eba3514b47a2fa841358454342"><strong id="mrs_01_1814__aba9fb7bb776346d8a3aeeeccee1188bb">vi job.properties</strong></p>
|
|
<p id="mrs_01_1814__a8f8cfb46c2c24c45b7a75afcc79efba6">Perform the following modifications:</p>
|
|
<p id="mrs_01_1814__p934881516612">Change the value of <span class="parmname" id="mrs_01_1814__parmname11254125919257"><b>userName</b></span> to the name of the human-machine user who submits the job, for example, <span class="parmvalue" id="mrs_01_1814__parmvalue2025975962513"><b>userName=oozieuser</b></span>.</p>
|
|
</p></li><li id="mrs_01_1814__lcabd43b3fb314898bc19ded29c90a2b3"><span>Run the <strong id="mrs_01_1814__b2045982122618">oozie job</strong> command to run the workflow file:</span><p><p id="mrs_01_1814__p54529342154411"><strong id="mrs_01_1814__b154661143260">oozie job -oozie https://</strong><em id="mrs_01_1814__i12467245263">Host name of the Oozie role</em><strong id="mrs_01_1814__b204676412269">:21003/oozie/ -config job.properties -run</strong></p>
|
|
<div class="note" id="mrs_01_1814__nab1e7bedb6514357b2d857cdd437f4c6"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1814__u505e120ab7fc4d7dada573bbfaf0fb3a"><li id="mrs_01_1814__lb21c9804740942e0b20f979562530dfc">The command parameters are described as follows:<p id="mrs_01_1814__add0178af9cb74a7c8bc43ccde17e8056"><a name="mrs_01_1814__lb21c9804740942e0b20f979562530dfc"></a><a name="lb21c9804740942e0b20f979562530dfc"></a><strong id="mrs_01_1814__b740627192618">-oozie</strong> URL of the Oozie server that executes a job</p>
|
|
<p id="mrs_01_1814__ac71440e8b53147bcb42b441a9d6f02cf"><strong id="mrs_01_1814__b5494692263">-config</strong> Workflow property file</p>
|
|
<p id="mrs_01_1814__a80ac946821e648c8b67f4d19e7a48dc0"><strong id="mrs_01_1814__b78083104266">-run</strong> Executing a workflow</p>
|
|
</li><li id="mrs_01_1814__l8c662c9385fb4673b33ebe1261d2f945">If a job ID, for example, <span class="parmvalue" id="mrs_01_1814__parmvalue101681112102616"><b>job: 0000021-140222101051722-oozie-omm-W</b></span>, is displayed after the workflow file is executed, the job is successfully submitted. You can view the execution results on the Oozie management page.<p id="mrs_01_1814__a980a1632d7b44c7a819da7bd820778e9">Log in to the Oozie web UI at <strong id="mrs_01_1814__b1860091532614">https</strong>://<em id="mrs_01_1814__i146013155260">IP address of the Oozie role</em><strong id="mrs_01_1814__b146011215142619">:21003/oozie</strong> as user <strong id="mrs_01_1814__b18602121519262">oozieuser</strong>.</p>
|
|
<p id="mrs_01_1814__afb508a57fb504357a1e2a1483d95a2e0">On the Oozie web UI, you can view the submitted workflow information based on the job ID in the table on the page.</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
<p id="mrs_01_1814__p8060118"></p>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1812.html">Using Oozie Client to Submit an Oozie Job</a></div>
|
|
</div>
|
|
</div>
|
|
|