forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
41 lines
7.3 KiB
HTML
41 lines
7.3 KiB
HTML
<a name="mrs_01_1808"></a><a name="mrs_01_1808"></a>
|
|
|
|
<h1 class="topictitle1">Using Oozie from Scratch</h1>
|
|
<div id="body1596077979069"><p id="mrs_01_1808__a445c022e3bf64521954bec29e92ccf4d">Oozie is an open-source workflow engine that is used to schedule and coordinate Hadoop jobs.</p>
|
|
<p id="mrs_01_1808__ab0644f04f4884d20af810216ff873396">Oozie can be used to submit a wide array of jobs, such as Hive, Spark2x, Loader, MapReduce, Java, DistCp, Shell, HDFS, SSH, SubWorkflow, Streaming, and scheduled jobs.</p>
|
|
<p id="mrs_01_1808__p8060118">This section describes how to use the Oozie client to submit a MapReduce job.</p>
|
|
<div class="section" id="mrs_01_1808__section14867051154717"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_1808__p1860261613816">The client has been installed. For example, the installation directory is <strong id="mrs_01_1808__b3287174517612">/opt</strong><strong id="mrs_01_1808__b1428715452617"></strong><strong id="mrs_01_1808__b628711453613">/client</strong>. The client directory in the following operations is only an example. Change it based on the actual installation directory onsite.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1808__section1882381914"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1808__ol233319318216"><li id="mrs_01_1808__l6109f1763aad473e9edba417b410db5b"><span>Log in to the node where the client is installed as the client installation user.</span></li><li id="mrs_01_1808__l1cfc094e3573492b8624ed13e10e00f5"><span>Run the following command to go to the client installation directory, for example, <strong id="mrs_01_1808__b089413965610">/opt/Bigdata/client</strong>:</span><p><p id="mrs_01_1808__a42ca79399e7d4e2fa6891cfe15b49a06"><strong id="mrs_01_1808__a5794dab6880b412f86fe90a137992e5b">cd /opt/client</strong></p>
|
|
</p></li><li id="mrs_01_1808__ldc4dc92d64ef41a78f0e483a058f6147"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_1808__a0f7284074ef6400eac7ed1a745a3ec6d"><strong id="mrs_01_1808__abfe8c835946f4d87855400af0df9a1e3">source bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1808__lc10513c57e21483ab2a387e67e6f75d2"><span>Check the cluster authentication mode.</span><p><ul id="mrs_01_1808__u67315135e30d4f01b0f494e43d068f26"><li id="mrs_01_1808__lb3ba529a419944b09a3fb14b13c05ec4">If the cluster is in security mode, run the following command to authenticate the user: <em id="mrs_01_1808__i15910131381019">UserOozie</em> indicates the user who submits tasks.<p id="mrs_01_1808__a4da6b34e80fa48fd884a26cbcc0a01d5"><strong id="mrs_01_1808__a018e75eef98c47e6badc55c9dbe1cdc7">kinit</strong> <em id="mrs_01_1808__i18960725712">User</em><em id="mrs_01_1808__i89601321574">Oozie</em></p>
|
|
</li><li id="mrs_01_1808__l5b95727d0d1e4588980e699579974e31">If the cluster is in normal mode, go to <a href="#mrs_01_1808__li1803152871315">5</a>.</li></ul>
|
|
</p></li><li id="mrs_01_1808__li1803152871315"><a name="mrs_01_1808__li1803152871315"></a><a name="li1803152871315"></a><span>Upload the Oozie configuration file and JAR package to HDFS.</span><p><p id="mrs_01_1808__p3946938518147"><strong id="mrs_01_1808__b4106653818388">hdfs dfs -mkdir /user/</strong><em id="mrs_01_1808__i364170301826">User</em><em id="mrs_01_1808__i11736592320">Oozie</em></p>
|
|
<p id="mrs_01_1808__p2079044118221"><strong id="mrs_01_1808__b438718500466">hdfs dfs -put -f /opt/client/Oozie/oozie-client-</strong>*<strong id="mrs_01_1808__b63871850184614">/examples /user/</strong><em id="mrs_01_1808__i1125811614313">User</em><em id="mrs_01_1808__i1425813165312">Oozie</em>/</p>
|
|
<div class="note" id="mrs_01_1808__note192214710317"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1808__ul1962208173216"><li id="mrs_01_1808__li3962208153217"><strong id="mrs_01_1808__b1716441216574">/opt</strong><strong id="mrs_01_1808__b1516141835714">/client/</strong> is an example client installation directory. Change it to the actual installation directory.</li><li id="mrs_01_1808__li645319422329"><strong id="mrs_01_1808__b0393648165711">UserOozie</strong> indicates the name of the user who submits jobs.</li></ul>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_1808__li24851666115553"><span>Run the following commands to modify the job execution configuration file:</span><p><p id="mrs_01_1808__p22338403115553"><strong id="mrs_01_1808__b2377111134720">cd /opt/client/Oozie/oozie-client-</strong>*<strong id="mrs_01_1808__b1237771116476">/examples/apps/map-reduce/</strong></p>
|
|
<p id="mrs_01_1808__p66827906115553"><strong id="mrs_01_1808__b59112663151320">vi job.properties</strong></p>
|
|
<pre class="screen" id="mrs_01_1808__screen599219511444"><strong id="mrs_01_1808__b73665204124">nameNode=hdfs://hacluster</strong>
|
|
<strong id="mrs_01_1808__b7521119154010">resourceManager=</strong><strong id="mrs_01_1808__b35225974012"><em id="mrs_01_1808__i63617519115553">10.64.35.161</em></strong><strong id="mrs_01_1808__b1452217918404">:</strong><strong id="mrs_01_1808__b5522159194017"></strong><strong id="mrs_01_1808__b16522890403">8032</strong> (<strong id="mrs_01_1808__b4803131119563">10.64.35.161</strong> is the service plane IP address of the Yarn resourceManager (active) node, and 8032 is the port number of <strong id="mrs_01_1808__b1480471145615">yarn.resourcemanager.port</strong>)
|
|
<strong id="mrs_01_1808__b17973524161215">queueName=default</strong>
|
|
<strong id="mrs_01_1808__b283727101214">examplesRoot=examples</strong>
|
|
<strong id="mrs_01_1808__b75472299129">user.name=admin</strong>
|
|
<strong id="mrs_01_1808__b985953371211">oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce# HDFS upload path</strong>
|
|
outputDir=map-reduce
|
|
oozie.wf.rerun.failnodes=true</pre>
|
|
</p></li><li id="mrs_01_1808__li13330247115553"><span>Run the following command to execute the Oozie job:</span><p><p id="mrs_01_1808__p52863363115553"><strong id="mrs_01_1808__b4708420132">oozie job -oozie https://</strong><em id="mrs_01_1808__i19718491313">Host name of the Oozie role</em><strong id="mrs_01_1808__b97110491316">:21003/oozie/ -config job.properties -run</strong></p>
|
|
<pre class="screen" id="mrs_01_1808__screen714121313596">[root@kwephispra44947 map-reduce]# oozie job -oozie https://kwephispra44948:21003/oozie/ -config job.properties -run
|
|
......
|
|
job: 0000000-200730163829770-oozie-omm-W</pre>
|
|
</p></li><li id="mrs_01_1808__li217311115252"><span>Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager (MRS 3.x or Later)</a>.</span></li><li id="mrs_01_1808__li1344541512516"><span>Choose <strong id="mrs_01_1808__b130510565577">Cluster</strong> > <em id="mrs_01_1808__i122975593575">Name of the desired cluster</em> > <strong id="mrs_01_1808__b2096284135815">Services</strong> > <strong id="mrs_01_1808__b148832765817">Oozie</strong>, click the hyperlink next to <strong id="mrs_01_1808__b1086081411584">Oozie WebUI</strong> to go to the Oozie page, and view the task execution result on the Oozie web UI.</span><p><div class="fignone" id="mrs_01_1808__fig128811421154318"><span class="figcap"><b>Figure 1 </b>Task execution result</span><br><span><img id="mrs_01_1808__image11195195444217" src="en-us_image_0000001387862162.png"></span></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1807.html">Using Oozie</a></div>
|
|
</div>
|
|
</div>
|
|
|