forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
40 lines
8.9 KiB
HTML
40 lines
8.9 KiB
HTML
<a name="mrs_01_1808"></a><a name="mrs_01_1808"></a>
|
|
|
|
<h1 class="topictitle1">Using Oozie from Scratch</h1>
|
|
<div id="body8662426"><p id="mrs_01_1808__en-us_topic_0000001173471274_a445c022e3bf64521954bec29e92ccf4d">Oozie is an open-source workflow engine that is used to schedule and coordinate Hadoop jobs.</p>
|
|
<p id="mrs_01_1808__en-us_topic_0000001173471274_ab0644f04f4884d20af810216ff873396">Oozie can be used to submit a wide array of jobs, such as Hive, Spark2x, Loader, MapReduce, Java, DistCp, Shell, HDFS, SSH, SubWorkflow, Streaming, and scheduled jobs.</p>
|
|
<p id="mrs_01_1808__en-us_topic_0000001173471274_p8060118">This section describes how to use the Oozie client to submit a MapReduce job.</p>
|
|
<div class="section" id="mrs_01_1808__en-us_topic_0000001173471274_section14867051154717"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_1808__en-us_topic_0000001173471274_p1860261613816">The client has been installed. For example, the installation directory is <strong id="mrs_01_1808__en-us_topic_0000001173471274_b3287174517612">/opt</strong><strong id="mrs_01_1808__en-us_topic_0000001173471274_b1428715452617"></strong><strong id="mrs_01_1808__en-us_topic_0000001173471274_b628711453613">/client</strong>. The client directory in the following operations is only an example. Change it based on the actual installation directory onsite.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1808__en-us_topic_0000001173471274_section1882381914"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1808__en-us_topic_0000001173471274_ol233319318216"><li id="mrs_01_1808__en-us_topic_0000001173471274_l6109f1763aad473e9edba417b410db5b"><span>Log in to the node where the client is installed as the client installation user.</span></li><li id="mrs_01_1808__en-us_topic_0000001173471274_l1cfc094e3573492b8624ed13e10e00f5"><span>Run the following command to go to the client installation directory. Assume that the client is installed in <strong id="mrs_01_1808__en-us_topic_0000001173471274_b1494811402543">/opt</strong><strong id="mrs_01_1808__en-us_topic_0000001173471274_b2948204095410"></strong><strong id="mrs_01_1808__en-us_topic_0000001173471274_b11948440135411">/client</strong>.</span><p><p id="mrs_01_1808__en-us_topic_0000001173471274_a42ca79399e7d4e2fa6891cfe15b49a06"><strong id="mrs_01_1808__en-us_topic_0000001173471274_a5794dab6880b412f86fe90a137992e5b">cd /opt/client</strong></p>
|
|
</p></li><li id="mrs_01_1808__en-us_topic_0000001173471274_ldc4dc92d64ef41a78f0e483a058f6147"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_1808__en-us_topic_0000001173471274_a0f7284074ef6400eac7ed1a745a3ec6d"><strong id="mrs_01_1808__en-us_topic_0000001173471274_abfe8c835946f4d87855400af0df9a1e3">source bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1808__en-us_topic_0000001173471274_lc10513c57e21483ab2a387e67e6f75d2"><span>Check the cluster authentication mode.</span><p><ul id="mrs_01_1808__en-us_topic_0000001173471274_u67315135e30d4f01b0f494e43d068f26"><li id="mrs_01_1808__en-us_topic_0000001173471274_lb3ba529a419944b09a3fb14b13c05ec4">If the cluster is in security mode, run the following command to authenticate the user: <em id="mrs_01_1808__en-us_topic_0000001173471274_i15910131381019">UserOozie</em> indicates the user who submits tasks.<p id="mrs_01_1808__en-us_topic_0000001173471274_a4da6b34e80fa48fd884a26cbcc0a01d5"><strong id="mrs_01_1808__en-us_topic_0000001173471274_a018e75eef98c47e6badc55c9dbe1cdc7">kinit</strong> <em id="mrs_01_1808__en-us_topic_0000001173471274_i18960725712">User</em><em id="mrs_01_1808__en-us_topic_0000001173471274_i89601321574">Oozie</em></p>
|
|
</li><li id="mrs_01_1808__en-us_topic_0000001173471274_l5b95727d0d1e4588980e699579974e31">If the cluster is in normal mode, go to <a href="#mrs_01_1808__en-us_topic_0000001173471274_li1803152871315">5</a>.</li></ul>
|
|
</p></li><li id="mrs_01_1808__en-us_topic_0000001173471274_li1803152871315"><a name="mrs_01_1808__en-us_topic_0000001173471274_li1803152871315"></a><a name="en-us_topic_0000001173471274_li1803152871315"></a><span>Upload the Oozie configuration file and JAR package to HDFS.</span><p><p id="mrs_01_1808__en-us_topic_0000001173471274_p3946938518147"><strong id="mrs_01_1808__en-us_topic_0000001173471274_b4106653818388">hdfs dfs -mkdir /user/</strong><em id="mrs_01_1808__en-us_topic_0000001173471274_i364170301826">User</em><em id="mrs_01_1808__en-us_topic_0000001173471274_i11736592320">Oozie</em></p>
|
|
<p id="mrs_01_1808__en-us_topic_0000001173471274_p2079044118221"><strong id="mrs_01_1808__en-us_topic_0000001173471274_b56956942183813">hdfs dfs -put -f /opt/client/Oozie/oozie-client-*/examples /user/</strong><em id="mrs_01_1808__en-us_topic_0000001173471274_i1125811614313">User</em><em id="mrs_01_1808__en-us_topic_0000001173471274_i1425813165312">Oozie</em>/</p>
|
|
<p id="mrs_01_1808__en-us_topic_0000001173471274_p1346162318488"><em id="mrs_01_1808__en-us_topic_0000001173471274_i7936111715193">UserOozie</em> indicates the user who submits tasks.</p>
|
|
</p></li><li id="mrs_01_1808__en-us_topic_0000001173471274_li24851666115553"><span>Run the following commands to modify the job execution configuration file:</span><p><p id="mrs_01_1808__en-us_topic_0000001173471274_p22338403115553"><strong id="mrs_01_1808__en-us_topic_0000001173471274_b58340299151316">cd /opt/client/Oozie/oozie-client-*/examples/apps/map-reduce/</strong></p>
|
|
<p id="mrs_01_1808__en-us_topic_0000001173471274_p66827906115553"><strong id="mrs_01_1808__en-us_topic_0000001173471274_b59112663151320">vi job.properties</strong></p>
|
|
<pre class="screen" id="mrs_01_1808__en-us_topic_0000001173471274_screen599219511444"><strong id="mrs_01_1808__en-us_topic_0000001173471274_b73665204124">nameNode=hdfs://hacluster</strong>
|
|
<strong id="mrs_01_1808__en-us_topic_0000001173471274_b7521119154010">resourceManager=</strong><strong id="mrs_01_1808__en-us_topic_0000001173471274_b35225974012"><em id="mrs_01_1808__en-us_topic_0000001173471274_i63617519115553">10.64.35.161</em></strong><strong id="mrs_01_1808__en-us_topic_0000001173471274_b1452217918404">:</strong><strong id="mrs_01_1808__en-us_topic_0000001173471274_b16522890403">8032</strong> (<strong id="mrs_01_1808__en-us_topic_0000001173471274_b4803131119563">10.64.35.161</strong> is the service plane IP address of the Yarn resourceManager (active) node, and 8032 is the port number of <strong id="mrs_01_1808__en-us_topic_0000001173471274_b1480471145615">yarn.resourcemanager.port</strong>)
|
|
<strong id="mrs_01_1808__en-us_topic_0000001173471274_b17973524161215">queueName=default</strong>
|
|
<strong id="mrs_01_1808__en-us_topic_0000001173471274_b283727101214">examplesRoot=examples</strong>
|
|
<strong id="mrs_01_1808__en-us_topic_0000001173471274_b75472299129">user.name=admin</strong>
|
|
<strong id="mrs_01_1808__en-us_topic_0000001173471274_b985953371211">oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce# HDFS upload path</strong>
|
|
outputDir=map-reduce
|
|
oozie.wf.rerun.failnodes=true</pre>
|
|
</p></li><li id="mrs_01_1808__en-us_topic_0000001173471274_li13330247115553"><span>Run the following command to execute the Oozie job:</span><p><p id="mrs_01_1808__en-us_topic_0000001173471274_p52863363115553"><strong id="mrs_01_1808__en-us_topic_0000001173471274_b4708420132">oozie job -oozie https://</strong><em id="mrs_01_1808__en-us_topic_0000001173471274_i19718491313">Host name of the Oozie role</em><strong id="mrs_01_1808__en-us_topic_0000001173471274_b97110491316">:21003/oozie/ -config job.properties -run</strong></p>
|
|
<pre class="screen" id="mrs_01_1808__en-us_topic_0000001173471274_screen714121313596">[root@kwephispra44947 map-reduce]# oozie job -oozie https://kwephispra44948:21003/oozie/ -config job.properties -run
|
|
......
|
|
job: 0000000-200730163829770-oozie-omm-W</pre>
|
|
</p></li><li id="mrs_01_1808__en-us_topic_0000001173471274_li217311115252"><span>Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>.</span></li><li id="mrs_01_1808__en-us_topic_0000001173471274_li1344541512516"><span>Choose <strong id="mrs_01_1808__en-us_topic_0000001173471274_b130510565577">Cluster</strong> > <em id="mrs_01_1808__en-us_topic_0000001173471274_i122975593575">Name of the desired cluster</em> > <strong id="mrs_01_1808__en-us_topic_0000001173471274_b2096284135815">Services</strong> > <strong id="mrs_01_1808__en-us_topic_0000001173471274_b148832765817">Oozie</strong>, click the hyperlink next to <strong id="mrs_01_1808__en-us_topic_0000001173471274_b1086081411584">Oozie WebUI</strong> to go to the Oozie page, and view the task execution result on the Oozie web UI.</span><p><div class="fignone" id="mrs_01_1808__en-us_topic_0000001173471274_fig74611263101"><span class="figcap"><b>Figure 1 </b>Task execution result</span><br><span><img id="mrs_01_1808__en-us_topic_0000001173471274_image1646113265103" src="en-us_image_0000001349139549.png"></span></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1807.html">Using Oozie</a></div>
|
|
</div>
|
|
</div>
|
|
|