forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
22 lines
5.9 KiB
HTML
22 lines
5.9 KiB
HTML
<a name="mrs_01_1821"></a><a name="mrs_01_1821"></a>
|
|
|
|
<h1 class="topictitle1">Submitting a Spark2x Job</h1>
|
|
<div id="body1591347171855"><div class="section" id="mrs_01_1821__seb13f36961e14490a77b16dde78b4386"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1821__a349ffe7f76764de189d08f87eaa568ef">This section describes how to submit an Oozie job of the Spark2x type on Hue.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1821__section12243121513"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1821__o3cfb8bdbdad542f89b2daa670dad85d4"><li id="mrs_01_1821__l45e0dd036f754c1493bec3220efb8b62"><span>Create a workflow. For details, see <a href="mrs_01_1818.html">Creating a Workflow</a>.</span></li><li id="mrs_01_1821__l98de8002e49143469b8a71cde3646029"><span>On the workflow editing page, select <span><img id="mrs_01_1821__i32de5183e74848a99b1e7f39df3f968d" src="en-us_image_0000001296250232.jpg"></span> next to <strong id="mrs_01_1821__b1858522413">Spark program</strong> and drag it to the operation area.</span></li><li id="mrs_01_1821__li139748715411"><span>In the Spark window that is displayed, set the value of <span class="parmname" id="mrs_01_1821__parmname3117194519"><b>Files</b></span>, for example, to <span class="parmvalue" id="mrs_01_1821__parmvalue76124210540"><b>hdfs://hacluster/user/admin/examples/apps/spark2x/lib/oozie-examples.jar</b></span>. Set the value of <span class="parmname" id="mrs_01_1821__parmname13605230525"><b>jar/py name</b></span>, for example, to <strong id="mrs_01_1821__b17724642134317">org.apache.oozie.example.SparkFileCopy</strong>, and click <strong id="mrs_01_1821__b34413194416">Add</strong>.</span></li><li id="mrs_01_1821__li12837441123820"><span>Set the value of <strong id="mrs_01_1821__b1843012273337">Main class</strong>, for example, <span class="parmvalue" id="mrs_01_1821__parmvalue186313551145"><b>org.apache.oozie.example.SparkFileCopy</b></span>.</span></li><li id="mrs_01_1821__l910935cc2a084236b8de28d7316e4930"><span>Click <strong id="mrs_01_1821__b2087010468918">PARAMETER+</strong> to add related input and output parameters.</span><p><p id="mrs_01_1821__ab4b6d5082943457ba102aa1b8365373a">For example, add the following parameters:</p>
|
|
<ul id="mrs_01_1821__ul177552555422"><li id="mrs_01_1821__li10757145511424"><strong id="mrs_01_1821__b817631810514">hdfs://hacluster/user/admin/examples/input-data/text/data.txt</strong></li><li id="mrs_01_1821__li29851457104213"><strong id="mrs_01_1821__b1369510564911">hdfs://hacluster/user/admin/examples/output-data/spark_workflow</strong></li></ul>
|
|
</p></li><li id="mrs_01_1821__li1652994117417"><span>In the <span class="wintitle" id="mrs_01_1821__wintitle69931947141110"><b>Options list</b></span> text box, specify Spark parameters, for example, <strong id="mrs_01_1821__b18755824593">--conf spark.yarn.archive=hdfs://hacluster/user/spark2x/jars/<span id="mrs_01_1821__text41611050163115">8.1.0.1</span>/spark-archive-2x.zip --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://hacluster/spark2xJobHistory2x</strong>.</span><p><div class="note" id="mrs_01_1821__note9425192413176"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1821__p34264242171"><span id="mrs_01_1821__text1975894211364">The version 8.1.0.1 is used as an example. Replace it with the actual version number.</span></p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_1821__li16249176864"><span>Click the configuration button <span><img id="mrs_01_1821__i74b00bf8902d46e395f4b70016a0af31" src="en-us_image_0000001295770796.jpg"></span> in the upper right corner. Set the value of <span class="parmname" id="mrs_01_1821__parmname1490015330713"><b>Spark Master</b></span>, for example, to <strong id="mrs_01_1821__b8926015106">yarn-cluster</strong>. Set the value of <span class="parmname" id="mrs_01_1821__parmname1363113431478"><b>Mode</b></span>, for example, <strong id="mrs_01_1821__b86291417104">cluster</strong>.</span></li><li id="mrs_01_1821__li7602121109"><span>On the configuration page that is displayed, click <span class="uicontrol" id="mrs_01_1821__uicontrol450216151320"><b>Delete +</b></span> to delete a directory, for example,<strong id="mrs_01_1821__b1650317151238"> hdfs://hacluster/user/admin/examples/output-data/spark_workflow</strong>.</span></li><li id="mrs_01_1821__li1464013210617"><span>Click <span class="uicontrol" id="mrs_01_1821__uicontrol13468713241"><b>PROPERTIES+</b></span> and add <strong id="mrs_01_1821__b104694131243">sharelib</strong> used by Oozie. Enter the attribute name <span class="parmname" id="mrs_01_1821__parmname84701013942"><b>oozie.action.sharelib.for.spark</b></span> in the left text box and the attribute value <span class="parmvalue" id="mrs_01_1821__parmvalue184713131745"><b>spark2x</b></span> in the right text box.</span></li><li id="mrs_01_1821__l64df422e4ba743b4a4d5ee3950778f05"><span>Click <span><img id="mrs_01_1821__image1236335111194" src="en-us_image_0000001349090429.png"></span> in the upper right corner of the Oozie editor.</span><p><p id="mrs_01_1821__ad1272ef4806748bf868ee4c28bb8e9c6">If you need to modify the job name before saving the job (default value: <span class="parmname" id="mrs_01_1821__parmname81698351189"><b>My Workflow</b></span>), click the name directly for modification, for example, <strong id="mrs_01_1821__b61705351289">Spark-Workflow</strong>.</p>
|
|
</p></li><li id="mrs_01_1821__l24d79648d0bc4d16809f9aa9f8d36465"><span>After the configuration is saved, click <span><img id="mrs_01_1821__image75416412133" src="en-us_image_0000001349289909.jpg"></span>, and submit the job.</span><p><p id="mrs_01_1821__aaeaa7dfd7d84407391f84d425b70f0c7">After the job is submitted, you can view the related contents of the job, such as the detailed information, logs, and processes, on Hue.</p>
|
|
</p></li></ol>
|
|
</div>
|
|
<p id="mrs_01_1821__p8060118"></p>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1819.html">Submitting a Workflow Job</a></div>
|
|
</div>
|
|
</div>
|
|
|