forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
81 lines
17 KiB
HTML
81 lines
17 KiB
HTML
<a name="mrs_01_0473"></a><a name="mrs_01_0473"></a>
|
|
|
|
<h1 class="topictitle1">Using Flink from Scratch</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_0473__section11185017184917"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0473__en-us_topic_0264266269_p8060118">This section describes how to use Flink to run wordcount jobs.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0473__en-us_topic_0264266269_section148416033913"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_0473__en-us_topic_0264266269_ul16771121414013"><li id="mrs_01_0473__en-us_topic_0264266269_li10771151419019">Flink has been installed in the MRS cluster and all components in the cluster are running properly.</li><li id="mrs_01_0473__en-us_topic_0264266269_li1310817171007">The cluster client has been installed, for example, in the <strong id="mrs_01_0473__b52942057414">/opt/hadoopclient</strong> directory.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_0473__en-us_topic_0264266269_section286111359366"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_0473__en-us_topic_0264266269_ol8931473819"><li id="mrs_01_0473__en-us_topic_0264266269_li18755848125417"><span>Log in to the node where the client is installed as the client installation user.</span></li><li id="mrs_01_0473__li11744174191512"><span>Run the following commands to go to the client installation directory.</span><p><p id="mrs_01_0473__p1974556181712"><strong id="mrs_01_0473__b9710161117174">cd /opt/hadoopclient</strong></p>
|
|
</p></li><li id="mrs_01_0473__en-us_topic_0264266269_li055115014556"><span>Run the following command to initialize environment variables:</span><p><p id="mrs_01_0473__en-us_topic_0264266269_p526118239574"><strong id="mrs_01_0473__b27933045416">source /opt/hadoopclient/bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_0473__en-us_topic_0264266269_li7711161925717"><span>If Kerberos authentication is enabled for the cluster, perform the following substeps. If Kerberos authentication is not enabled for the cluster, skip the following substeps.</span><p><ol type="a" id="mrs_01_0473__en-us_topic_0264266269_ol332183617012"><li id="mrs_01_0473__en-us_topic_0264266269_li544855618118">Create a user, for example, <strong id="mrs_01_0473__b9798427104217">test</strong>, for submitting Flink jobs.<p id="mrs_01_0473__en-us_topic_0264266269_p19801727596">After a human-machine user is created, log in to FusionInsight Manager as the new user and change the initial password as prompted.</p>
|
|
<div class="note" id="mrs_01_0473__note0219155792911"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0473__en-us_topic_0264266269_p12247143015119">To submit or run jobs on Flink, the user must have the following permissions:</p>
|
|
<ul id="mrs_01_0473__ul87669116301"><li id="mrs_01_0473__en-us_topic_0264266269_li124833018119">If Ranger authentication is enabled, the current user must belong to the <strong id="mrs_01_0473__en-us_topic_0264266269_b68861779173">hadoop</strong> group or the user has been granted the <strong id="mrs_01_0473__en-us_topic_0264266269_b1736101717179">/flink</strong> read and write permissions in Ranger.</li><li id="mrs_01_0473__li9767018302">If Ranger authentication is disabled, the current user must belong to the <strong id="mrs_01_0473__b8916201019364">hadoop</strong> group.</li></ul>
|
|
</div></div>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li27421727143610">Log in to FusionInsight Manager and choose <strong id="mrs_01_0473__b1827717120314">System > Permission > User</strong>. On the displayed page, locate the row that contains the added user, click <strong id="mrs_01_0473__b1484189153115">More</strong> in the <strong id="mrs_01_0473__b10892514183111">Operation</strong> column, and select <strong id="mrs_01_0473__b24142183114">Download Authentication Credential</strong> to download the authentication credential file of the user to the local PC and decompress the file.</li><li id="mrs_01_0473__en-us_topic_0264266269_li68525322019">Copy the decompressed <strong id="mrs_01_0473__b167427551411">user.keytab</strong> and <strong id="mrs_01_0473__b8749055114112">krb5.conf</strong> files to the <strong id="mrs_01_0473__b2750155584119">/opt/hadoopclient/Flink/flink/conf</strong> directory on the client node.</li><li id="mrs_01_0473__en-us_topic_0264266269_li1214133623520">Log in to the client node and add the service IP address of the client node and the floating IP address of FusionInsight Manager to the <strong id="mrs_01_0473__b12118518114219">jobmanager.web.allow-access-address</strong> configuration item in the <strong id="mrs_01_0473__b12124618134219">/opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong> file. Use commas (,) to separate the IP addresses.<p id="mrs_01_0473__p65841864287"><strong id="mrs_01_0473__b629811138281">vi /opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong></p>
|
|
</li><li id="mrs_01_0473__li18481113415581">Configure security authentication.<p id="mrs_01_0473__p698423465818"><a name="mrs_01_0473__li18481113415581"></a><a name="li18481113415581"></a>Add the <strong id="mrs_01_0473__b8781175815421">keytab</strong> path and username to the <strong id="mrs_01_0473__b07815586426">/opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong> configuration file.</p>
|
|
<pre class="screen" id="mrs_01_0473__screen0761164315810"><strong id="mrs_01_0473__b633223812614">security.kerberos.login.keytab: </strong><em id="mrs_01_0473__i2015203113616"><user.keytab file path></em>
|
|
<strong id="mrs_01_0473__b1496214814610">security.kerberos.login.principal: </strong><em id="mrs_01_0473__i6474346468"><Username></em></pre>
|
|
<p id="mrs_01_0473__en-us_topic_0264266269_p2340691346">Example:</p>
|
|
<pre class="screen" id="mrs_01_0473__screen1771775812583">security.kerberos.login.keytab: /opt/hadoopclient/Flink/flink/conf/user.keytab
|
|
security.kerberos.login.principal: test</pre>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li16702185971011">Configure security hardening by referring to <a href="mrs_01_1583.html">Authentication and Encryption</a>. Run the following commands to set a password for submitting jobs.<p id="mrs_01_0473__p157041740961"><strong id="mrs_01_0473__b13348162317714">cd /opt/hadoopclient/Flink/flink/bin</strong></p>
|
|
<p id="mrs_01_0473__en-us_topic_0264266269_p127132597108"><strong id="mrs_01_0473__en-us_topic_0264266269_b1967818111713">sh generate_keystore.sh</strong></p>
|
|
<p id="mrs_01_0473__en-us_topic_0264266269_p5713165991012">The script automatically changes the SSL-related parameter values in the <strong id="mrs_01_0473__b846916854311">/opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong> file.</p>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li173581132005">Configure paths for the client to access the <strong id="mrs_01_0473__b713713613436">flink.keystore</strong> and <strong id="mrs_01_0473__b181371336104310">flink.truststore</strong> files.<ul id="mrs_01_0473__en-us_topic_0264266269_ul955713364211"><li id="mrs_01_0473__li17301647712">Absolute path<p id="mrs_01_0473__p36889477117"><a name="mrs_01_0473__li17301647712"></a><a name="li17301647712"></a>After the <strong id="mrs_01_0473__b16613125013599">generate_keystore.sh</strong> script is executed, the <strong id="mrs_01_0473__b13635145635911">flink.keystore</strong> and <strong id="mrs_01_0473__b188059017014">flink.truststore</strong> file paths are automatically set to absolute paths in the <strong id="mrs_01_0473__b18989136809">flink-conf.yaml</strong> file by default. In this case, you need to place the <strong id="mrs_01_0473__b61501023364">flink.keystore</strong> and <strong id="mrs_01_0473__b1041972615620">flink.truststore</strong> files in the <strong id="mrs_01_0473__b1025319281768">conf</strong> directory to the absolute paths of the Flink client and each Yarn node, respectively.</p>
|
|
</li><li id="mrs_01_0473__li03965913418">Relative path (recommended)<div class="p" id="mrs_01_0473__p1112112107411"><a name="mrs_01_0473__li03965913418"></a><a name="li03965913418"></a>Perform the following steps to set the file paths of <strong id="mrs_01_0473__b390241611310">flink.keystore</strong> and <strong id="mrs_01_0473__b49085161333">flink.truststore</strong> to relative paths and ensure that the directory where the Flink client command is executed can directly access the relative paths.<ol class="substepthirdol" id="mrs_01_0473__en-us_topic_0264266269_ol870917381216"><li id="mrs_01_0473__en-us_topic_0264266269_li10709238625">Create a directory, for example, <strong id="mrs_01_0473__b77491119104319">ssl</strong>, in <strong id="mrs_01_0473__b157571619174316">/opt/hadoopclient/Flink/flink/conf/</strong>.<p id="mrs_01_0473__p1178520121858"><strong id="mrs_01_0473__b153011525151">cd /opt/hadoopclient/Flink/flink/conf/</strong></p>
|
|
<p id="mrs_01_0473__p14454613550"><strong id="mrs_01_0473__b1232732511510">mkdir ssl</strong></p>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li1170933811211">Move the <strong id="mrs_01_0473__b1997115104419">flink.keystore</strong> and <strong id="mrs_01_0473__b2214141317411">flink.truststore</strong> files to the new paths.<p id="mrs_01_0473__p1385032915261"><strong id="mrs_01_0473__b271064802610">mv flink.keystore ssl/</strong></p>
|
|
<p id="mrs_01_0473__p166719381262"><strong id="mrs_01_0473__b5729184892619">mv flink.truststore ssl/</strong></p>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li1770914381627">Change the values of the following parameters to relative paths in the <strong id="mrs_01_0473__b78357315110">flink-conf.yaml</strong> file:<div class="p" id="mrs_01_0473__p1057032711285"><strong id="mrs_01_0473__b1541122682815">vi /opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong><pre class="screen" id="mrs_01_0473__screen1082382522810"><strong id="mrs_01_0473__b1611850155414">security.ssl.keystore: ssl/flink.keystore</strong>
|
|
<strong id="mrs_01_0473__b202231053155414">security.ssl.truststore: ssl/flink.truststore</strong></pre>
|
|
</div>
|
|
</li></ol>
|
|
</div>
|
|
</li></ul>
|
|
</li></ol>
|
|
</p></li><li id="mrs_01_0473__en-us_topic_0264266269_li43014216812"><span>Run a wordcount job.</span><p><ul id="mrs_01_0473__en-us_topic_0264266269_ul346719451784"><li id="mrs_01_0473__en-us_topic_0264266269_li325519443332">Normal cluster (Kerberos authentication disabled)<ul id="mrs_01_0473__en-us_topic_0264266269_ul7255244123310"><li id="mrs_01_0473__en-us_topic_0264266269_li4255444163312">Run the following commands to start a session and submit a job in the session:<p id="mrs_01_0473__p1712132583010"><a name="mrs_01_0473__en-us_topic_0264266269_li4255444163312"></a><a name="en-us_topic_0264266269_li4255444163312"></a><strong id="mrs_01_0473__b17144143133015">yarn-session.sh -nm "</strong><em id="mrs_01_0473__i5145103114303">session-name</em><strong id="mrs_01_0473__b17144131143016">"</strong></p>
|
|
<p id="mrs_01_0473__p1212525133015"><strong id="mrs_01_0473__b01216256306">flink run /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li7255114415335">Run the following command to submit a single job on Yarn:<p id="mrs_01_0473__p12178846143010"><a name="mrs_01_0473__en-us_topic_0264266269_li7255114415335"></a><a name="en-us_topic_0264266269_li7255114415335"></a><strong id="mrs_01_0473__b1417815460303">flink run -m yarn-cluster /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li></ul>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li14467945786">Security cluster (Kerberos authentication enabled)<ul id="mrs_01_0473__en-us_topic_0264266269_ul185369348912"><li id="mrs_01_0473__en-us_topic_0264266269_li13754134017107">If the <strong id="mrs_01_0473__en-us_topic_0264266269_b524013611911">flink.keystore</strong> and <strong id="mrs_01_0473__en-us_topic_0264266269_b132451961298">flink.truststore</strong> file paths are relative paths:<ul id="mrs_01_0473__en-us_topic_0264266269_ul1477219408108"><li id="mrs_01_0473__en-us_topic_0264266269_li7771174051014">Run the following command in the directory at the same level as <strong id="mrs_01_0473__b29381318518">ssl</strong> to start the session and submit the job in the session. <strong id="mrs_01_0473__b16631423259">ssl/</strong> is a relative path.<p id="mrs_01_0473__p4775142183217"><strong id="mrs_01_0473__b121071150123214">cd /opt/hadoopclient/Flink/flink/conf/</strong></p>
|
|
<p id="mrs_01_0473__p141888063216"><strong id="mrs_01_0473__b14862113336">yarn-session.sh -t ssl/ -nm "</strong><em id="mrs_01_0473__i17501511133311">session-name</em><strong id="mrs_01_0473__b54863114336">"</strong></p>
|
|
<pre class="screen" id="mrs_01_0473__screen181211826145612">...
|
|
Cluster started: Yarn cluster with application id application_1624937999496_0017
|
|
JobManager Web Interface: http://192.168.1.150:32261
|
|
</pre>
|
|
<p id="mrs_01_0473__p116262152418">Start a new client connection and submit the job:</p>
|
|
<p id="mrs_01_0473__p1965815331841"><strong id="mrs_01_0473__b191813119513">source /opt/hadoopclient/bigdata_env</strong></p>
|
|
<p id="mrs_01_0473__p10188305326"><strong id="mrs_01_0473__b4188190123213">flink run /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
<pre class="screen" id="mrs_01_0473__screen380315197589">...
|
|
Job has been submitted with JobID 587d5498fff18d8b2501fdf7ebb9c4fb
|
|
Program execution finished
|
|
Job with JobID 587d5498fff18d8b2501fdf7ebb9c4fb has finished.
|
|
Job Runtime: 19917 ms</pre>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li87721640181013">Run the following command to submit a single job on Yarn:<p id="mrs_01_0473__p3434111714534"><a name="mrs_01_0473__en-us_topic_0264266269_li87721640181013"></a><a name="en-us_topic_0264266269_li87721640181013"></a><strong id="mrs_01_0473__b1191318155310">cd /opt/hadoopclient/Flink/flink/conf/</strong></p>
|
|
<p id="mrs_01_0473__p2039154053313"><strong id="mrs_01_0473__b1239114063315">flink run -m yarn-cluster -yt ssl/ /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
<pre class="screen" id="mrs_01_0473__screen8550185414549">...
|
|
Cluster started: Yarn cluster with application id application_1624937999496_0016
|
|
Job has been submitted with JobID e9c59fb48f44feae7b62dd90336d6d7f
|
|
Program execution finished
|
|
Job with JobID e9c59fb48f44feae7b62dd90336d6d7f has finished.
|
|
Job Runtime: 18155 ms</pre>
|
|
</li></ul>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li7855183017911">If the <strong id="mrs_01_0473__b19979153113211">flink.keystore</strong> and <strong id="mrs_01_0473__b19851314218">flink.truststore</strong> file paths are absolute paths:<ul id="mrs_01_0473__en-us_topic_0264266269_ul1996520345101"><li id="mrs_01_0473__en-us_topic_0264266269_li3965334161017">Run the following commands to start a session and submit a job in the session:<p id="mrs_01_0473__p91291415619"><a name="mrs_01_0473__en-us_topic_0264266269_li3965334161017"></a><a name="en-us_topic_0264266269_li3965334161017"></a><strong id="mrs_01_0473__b4643044610">cd /opt/hadoopclient/Flink/flink/conf/</strong></p>
|
|
<p id="mrs_01_0473__p1344849183112"><strong id="mrs_01_0473__b19538714133112">yarn-session.sh -nm "</strong><em id="mrs_01_0473__i75581214173112">session-name</em><strong id="mrs_01_0473__b1053910144313">"</strong></p>
|
|
<p id="mrs_01_0473__p1244839113110"><strong id="mrs_01_0473__b2448396318">flink run /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li><li id="mrs_01_0473__en-us_topic_0264266269_li149651734171016">Run the following command to submit a single job on Yarn:<p id="mrs_01_0473__p1663142143113"><a name="mrs_01_0473__en-us_topic_0264266269_li149651734171016"></a><a name="en-us_topic_0264266269_li149651734171016"></a><strong id="mrs_01_0473__b1066310212317">flink run -m yarn-cluster /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li></ul>
|
|
</li></ul>
|
|
</li></ul>
|
|
</p></li><li id="mrs_01_0473__en-us_topic_0264266269_li09212438188"><span>Log in to FusionInsight Manager as a running user, go to the native page of the Yarn service, find the application of the corresponding job, and click the application name to go to the job details page.</span><p><ul id="mrs_01_0473__en-us_topic_0264266269_ul19954161472312"><li id="mrs_01_0473__en-us_topic_0264266269_li1595421420238">If the job is not completed, click <strong id="mrs_01_0473__b18904103212216">Tracking URL</strong> to go to the native Flink page and view the job running information.</li><li id="mrs_01_0473__en-us_topic_0264266269_li495411417238">If the job submitted in a session has been completed, you can click <strong id="mrs_01_0473__b1832682913413">Tracking URL</strong> to log in to the native Flink service page to view job information.<div class="fignone" id="mrs_01_0473__en-us_topic_0264266269_fig1043856121716"><span class="figcap"><b>Figure 1 </b>Application</span><br><span><img id="mrs_01_0473__en-us_topic_0264266269_image13740172014162" src="en-us_image_0000001349058841.png"></span></div>
|
|
</li></ul>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0591.html">Using Flink</a></div>
|
|
</div>
|
|
</div>
|
|
|