forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
119 lines
38 KiB
HTML
119 lines
38 KiB
HTML
<a name="mrs_01_0473"></a><a name="mrs_01_0473"></a>
|
|
|
|
<h1 class="topictitle1">Using Flink from Scratch</h1>
|
|
<div id="body1589421637992"><p id="mrs_01_0473__p8060118">This section describes how to use Flink to run wordcount jobs.</p>
|
|
<div class="section" id="mrs_01_0473__section148416033913"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_0473__ul16771121414013"><li id="mrs_01_0473__li10771151419019">Flink has been installed in an MRS cluster.</li><li id="mrs_01_0473__li1310817171007">The cluster runs properly and the client has been correctly installed, for example, in the <strong id="mrs_01_0473__b14933153513248">/opt/hadoopclient</strong> directory. The client directory in the following operations is only an example. Change it to the actual installation directory.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_0473__section286111359366"><h4 class="sectiontitle">Using the Flink Client (Versions Earlier Than MRS 3.x)</h4><ol id="mrs_01_0473__ol8931473819"><li id="mrs_01_0473__li18755848125417"><span>Log in to the node where the client is installed as the client installation user.</span></li><li id="mrs_01_0473__li11744174191512"><span>Run the following command to go to the client installation directory:</span><p><p id="mrs_01_0473__p1974556181712"><strong id="mrs_01_0473__b9710161117174">cd /opt/hadoopclient</strong></p>
|
|
</p></li><li id="mrs_01_0473__li055115014556"><span>Run the following command to initialize environment variables:</span><p><p id="mrs_01_0473__p526118239574"><b><span class="cmdname" id="mrs_01_0473__cmdname995743125717">source /opt/hadoopclient/bigdata_env</span></b></p>
|
|
</p></li><li id="mrs_01_0473__li7711161925717"><span>If Kerberos authentication is enabled for the cluster, perform the following steps. If not, skip this whole step.</span><p><ol type="a" id="mrs_01_0473__ol332183617012"><li id="mrs_01_0473__li544855618118">Prepare a user for submitting Flink jobs..</li><li id="mrs_01_0473__li27421727143610">Log in to Manager and download the authentication credential.<p id="mrs_01_0473__p06115411182"><a name="mrs_01_0473__li27421727143610"></a><a name="li27421727143610"></a>Log in to Manager of the cluster. For details, see <a href="mrs_01_0102.html">Accessing MRS Manager (Versions Earlier Than MRS 3.x)</a>. Choose <strong id="mrs_01_0473__b188561183252155">System Settings</strong> > <strong id="mrs_01_0473__b135296345952155">User Management</strong>. In the <strong id="mrs_01_0473__b132949543652155">Operation</strong> column of the row that contains the added user, choose <strong id="mrs_01_0473__b154032291852155">More</strong> > <strong id="mrs_01_0473__b17221830452155">Download Authentication Credential</strong>.</p>
|
|
</li><li id="mrs_01_0473__li68525322019">Decompress the downloaded authentication credential package and copy the <strong id="mrs_01_0473__b1314681634118">user.keytab</strong> file to the client node, for example, to the <span class="filepath" id="mrs_01_0473__filepath18301323919"><b>/opt/hadoopclient/Flink/flink/conf</b></span> directory on the client node. If the client is installed on a node outside the cluster, copy the <strong id="mrs_01_0473__b1085915504518">krb5.conf</strong> file to the <strong id="mrs_01_0473__b1541794417232">/etc/</strong> directory on this node.</li><li id="mrs_01_0473__li282116149110">Configure security authentication by adding the <strong id="mrs_01_0473__b1132133111291">keytab</strong> path and username in the <strong id="mrs_01_0473__b6327831182913">/opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong> configuration file.<p id="mrs_01_0473__p172103018310"><strong id="mrs_01_0473__b633223812614">security.kerberos.login.keytab: </strong><em id="mrs_01_0473__i2015203113616"><user.keytab file path></em></p>
|
|
<p id="mrs_01_0473__p14210701835"><strong id="mrs_01_0473__b1496214814610">security.kerberos.login.principal: </strong><em id="mrs_01_0473__i6474346468"><Username></em></p>
|
|
<p id="mrs_01_0473__p2340691346">Example:</p>
|
|
<p id="mrs_01_0473__p4441183614213">security.kerberos.login.keytab: /opt/hadoopclient/Flink/flink/conf/user.keytab</p>
|
|
<p id="mrs_01_0473__p17441143618211">security.kerberos.login.principal: test</p>
|
|
</li><li id="mrs_01_0473__li16702185971011">Generate the <strong id="mrs_01_0473__b10806164319543">generate_keystore.sh</strong> script and place it in the <strong id="mrs_01_0473__b815711517556">bin</strong> directory of the Flink client. In the <strong id="mrs_01_0473__b13530174152917">bin</strong> directory of the Flink client, run the following command to perform security hardening. For details, see <span id="mrs_01_0473__ph1294413550112"><a href="https://docs.otc.t-systems.com/cmpntguide/mrs/mrs_01_1583.html" target="_blank" rel="noopener noreferrer">Authentication and Encryption</a></span>. Set <strong id="mrs_01_0473__b16920155415375">password</strong> in the following command to a password for submitting jobs:<p id="mrs_01_0473__p127132597108"><strong id="mrs_01_0473__b1967818111713">sh generate_keystore.sh <<em id="mrs_01_0473__i2229191315177">password</em>></strong></p>
|
|
<p id="mrs_01_0473__p5713165991012">The script automatically replaces the SSL value in the <strong id="mrs_01_0473__b726875514297">/opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong> file. For an MRS 2.<em id="mrs_01_0473__i1516992417413">x</em> or earlier security cluster, external SSL is disabled by default. To enable external SSL, configure the parameter and run the script again. For details, see <span id="mrs_01_0473__ph118943810511"><a href="https://docs.otc.t-systems.com/cmpntguide/mrs/mrs_01_0594.html" target="_blank" rel="noopener noreferrer">Security Hardening</a></span>.</p>
|
|
<div class="note" id="mrs_01_0473__note93103595527"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0473__ul18746101312719"><li id="mrs_01_0473__li174751310272">You do not need to manually generate the <strong id="mrs_01_0473__b1526252253018">generate_keystore.sh</strong> script.</li><li id="mrs_01_0473__li974711372719">After authentication and encryption, the generated <strong id="mrs_01_0473__b16417144971810">flink.keystore</strong>, <strong id="mrs_01_0473__b174282049181814">flink.truststore</strong>, and <strong id="mrs_01_0473__b6429154981818">security.cookie</strong> items are automatically filled in the corresponding configuration items in <span class="filepath" id="mrs_01_0473__filepath94301649151813"><b>flink-conf.yaml</b></span>.</li></ul>
|
|
</div></div>
|
|
</li><li id="mrs_01_0473__li173581132005">Configure paths for the client to access the <strong id="mrs_01_0473__b109747502534">flink.keystore</strong> and <strong id="mrs_01_0473__b697510502536">flink.truststore</strong> files.<ul id="mrs_01_0473__ul955713364211"><li id="mrs_01_0473__li355853618212">Absolute path: After the script is executed, the file path of <strong id="mrs_01_0473__b145921314304">flink.keystore</strong> and <strong id="mrs_01_0473__b759213312304">flink.truststore</strong> is automatically set to the absolute path <strong id="mrs_01_0473__b1359333153011">/opt/hadoopclient/Flink/flink/conf/</strong> in the <strong id="mrs_01_0473__b459314318306">flink-conf.yaml</strong> file. In this case, you need to move the <strong id="mrs_01_0473__b1359315318305">flink.keystore</strong> and <strong id="mrs_01_0473__b1459312317304">flink.truststore</strong> files from the <strong id="mrs_01_0473__b159463117308">conf</strong> directory to this absolute path on the Flink client and Yarn nodes.</li><li id="mrs_01_0473__li370920381229">Relative path: Perform the following steps to set the file path of <strong id="mrs_01_0473__b12814141014575">flink.keystore</strong> and <strong id="mrs_01_0473__b168142010155715">flink.truststore</strong> to the relative path and ensure that the directory where the Flink client command is executed can directly access the relative paths.<ol class="substepthirdol" id="mrs_01_0473__ol870917381216"><li id="mrs_01_0473__li10709238625">Create a directory, for example, <strong id="mrs_01_0473__b488651116319">ssl</strong>, in <strong id="mrs_01_0473__b14886111111317">/opt/hadoopclient/Flink/flink/conf/</strong>.<p id="mrs_01_0473__p2058520491568"><strong id="mrs_01_0473__b12778913145711">cd /opt/hadoopclient/Flink/flink/conf/</strong></p>
|
|
<p id="mrs_01_0473__p152688575719"><strong id="mrs_01_0473__b19780111315572">mkdir ssl</strong></p>
|
|
</li><li id="mrs_01_0473__li1170933811211">Move the <strong id="mrs_01_0473__b19225104015319">flink.keystore</strong> and <strong id="mrs_01_0473__b823114033111">flink.truststore</strong> files to the <strong id="mrs_01_0473__b1723218401314">/opt/hadoopclient/Flink/flink/conf/ssl/</strong> directory.<p id="mrs_01_0473__p1074535205711"><strong id="mrs_01_0473__b35315345580">mv flink.keystore ssl/</strong></p>
|
|
<p id="mrs_01_0473__p64291255145720"><strong id="mrs_01_0473__b35371034115814">mv flink.truststore ssl/</strong></p>
|
|
</li><li id="mrs_01_0473__li151356562119">Change the values of the following parameters to relative paths in the <strong id="mrs_01_0473__b565619533318">flink-conf.yaml</strong> file:<pre class="screen" id="mrs_01_0473__screen14431621182016">security.ssl.internal.keystore: ssl/flink.keystore
|
|
security.ssl.internal.truststore: ssl/flink.truststore</pre>
|
|
</li></ol>
|
|
</li></ul>
|
|
</li></ol>
|
|
</p></li><li id="mrs_01_0473__li43014216812"><span>Run a wordcount job.</span><p><div class="notice" id="mrs_01_0473__note15492152974915"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="mrs_01_0473__p12247143015119">To submit or run jobs on Flink, the user must have the following permissions:</p>
|
|
<ul id="mrs_01_0473__ul92474309115"><li id="mrs_01_0473__li124833018119">If Ranger authentication is enabled, the current user must belong to the <strong id="mrs_01_0473__b68861779173">hadoop</strong> group or the user has been granted the <strong id="mrs_01_0473__b1736101717179">/flink</strong> read and write permissions in Ranger.</li><li id="mrs_01_0473__li0248133071118">If Ranger authentication is disabled, the current user must belong to the <strong id="mrs_01_0473__b4159202341912">hadoop</strong> group.</li></ul>
|
|
</div></div>
|
|
<ul id="mrs_01_0473__ul346719451784"><li id="mrs_01_0473__li325519443332">Normal cluster (Kerberos authentication disabled)<ul id="mrs_01_0473__ul7255244123310"><li id="mrs_01_0473__li4255444163312">Run the following commands to start a session and submit a job in the session:<p id="mrs_01_0473__p197831326101919"><a name="mrs_01_0473__li4255444163312"></a><a name="li4255444163312"></a><strong id="mrs_01_0473__b1334017352192">yarn-session.sh -nm "</strong><em id="mrs_01_0473__i1234353521911">session-name</em><strong id="mrs_01_0473__b83411735191911">"</strong></p>
|
|
<p id="mrs_01_0473__p1778312266195"><strong id="mrs_01_0473__b117822261198">flink run /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li><li id="mrs_01_0473__li7255114415335">Run the following command to submit a single job on Yarn:<p id="mrs_01_0473__p5565155191913"><a name="mrs_01_0473__li7255114415335"></a><a name="li7255114415335"></a><strong id="mrs_01_0473__b13565051101912">flink run -m yarn-cluster /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li></ul>
|
|
</li><li id="mrs_01_0473__li14467945786">Security cluster (Kerberos authentication enabled)<ul id="mrs_01_0473__ul185369348912"><li id="mrs_01_0473__li7855183017911">If the <strong id="mrs_01_0473__b78761822874">flink.keystore</strong> and <strong id="mrs_01_0473__b1788102217713">flink.truststore</strong> file are stored in the absolute path:<ul id="mrs_01_0473__ul1996520345101"><li id="mrs_01_0473__li3965334161017">Run the following commands to start a session and submit a job in the session:<p id="mrs_01_0473__p1076634618203"><a name="mrs_01_0473__li3965334161017"></a><a name="li3965334161017"></a><strong id="mrs_01_0473__b113965462016">yarn-session.sh -nm "</strong><em id="mrs_01_0473__i11411754142016">session-name</em><strong id="mrs_01_0473__b1613945412208">"</strong></p>
|
|
<p id="mrs_01_0473__p676594612208"><strong id="mrs_01_0473__b876584652014">flink run /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li><li id="mrs_01_0473__li149651734171016">Run the following command to submit a single job on Yarn:<p id="mrs_01_0473__p1910611112119"><a name="mrs_01_0473__li149651734171016"></a><a name="li149651734171016"></a><strong id="mrs_01_0473__b2010621172111">flink run -m yarn-cluster /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li></ul>
|
|
</li><li id="mrs_01_0473__li13754134017107">If the <strong id="mrs_01_0473__b524013611911">flink.keystore</strong> and <strong id="mrs_01_0473__b132451961298">flink.truststore</strong> files are stored in the relative path:<ul id="mrs_01_0473__ul1477219408108"><li id="mrs_01_0473__li7771174051014">In the same directory of SSL, run the following commands to start a session and submit jobs in the session. The SSL directory is a relative path. For example, if the SSL directory is <strong id="mrs_01_0473__b7193104516322">opt/hadoopclient/Flink/flink/conf/</strong>, then run the following commands in this directory:<p id="mrs_01_0473__p111826102213"><strong id="mrs_01_0473__b1256619397210">yarn-session.sh -t ssl/ -nm "</strong><em id="mrs_01_0473__i19584193972117">session-name</em><strong id="mrs_01_0473__b1456617398214">"</strong></p>
|
|
<p id="mrs_01_0473__p121821510102111"><strong id="mrs_01_0473__b1818212106216">flink run /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li><li id="mrs_01_0473__li87721640181013">Run the following command to submit a single job on Yarn:<p id="mrs_01_0473__p56611620152116"><a name="mrs_01_0473__li87721640181013"></a><a name="li87721640181013"></a><strong id="mrs_01_0473__b466122082112">flink run -m yarn-cluster -yt ssl/ /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li></ul>
|
|
</li></ul>
|
|
</li></ul>
|
|
</p></li><li id="mrs_01_0473__li17288428133"><span>After the job has been successfully submitted, the following information is displayed on the client:</span><p><div class="fignone" id="mrs_01_0473__fig7572041542"><span class="figcap"><b>Figure 1 </b>Job submitted successfully on Yarn</span><br><span><img id="mrs_01_0473__image1380945213307" src="en-us_image_0000001349289933.png"></span></div>
|
|
<div class="fignone" id="mrs_01_0473__fig2211144410227"><span class="figcap"><b>Figure 2 </b>Session started successfully</span><br><span><img id="mrs_01_0473__image6736162611818" src="en-us_image_0000001349289937.png"></span></div>
|
|
<div class="fignone" id="mrs_01_0473__fig1343995812714"><span class="figcap"><b>Figure 3 </b>Job submitted successfully in the session</span><br><span><img id="mrs_01_0473__image11803145571917" src="en-us_image_0000001295930780.png"></span></div>
|
|
</p></li><li id="mrs_01_0473__li09212438188"><span>Go to the native YARN service page, find the application of the job, and click the application name to go to the job details page. For details, see <span id="mrs_01_0473__ph63412418173"><a href="https://docs.otc.t-systems.com/cmpntguide/mrs/mrs_01_0784.html" target="_blank" rel="noopener noreferrer">Viewing Flink Job Information</a></span>.</span><p><ul id="mrs_01_0473__ul19954161472312"><li id="mrs_01_0473__li1595421420238">If the job is not completed, click <strong id="mrs_01_0473__b18904103212216">Tracking URL</strong> to go to the native Flink page and view the job running information.</li><li id="mrs_01_0473__li495411417238">If the job submitted in a session has been completed, you can click <strong id="mrs_01_0473__b2874121810599">Tracking URL</strong> to log in to the native Flink service page to view job information.<div class="fignone" id="mrs_01_0473__fig1043856121716"><span class="figcap"><b>Figure 4 </b>Application</span><br><span><img id="mrs_01_0473__image9297203618279" src="en-us_image_0000001439150893.png"></span></div>
|
|
</li></ul>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0473__section1160381713011"><h4 class="sectiontitle">Using the Flink Client (MRS 3.x or Later)</h4><ol id="mrs_01_0473__ol113322571308"><li id="mrs_01_0473__li1733395773011"><span>Log in to the node where the client is installed as the client installation user.</span></li><li id="mrs_01_0473__li152512427244"><span>Run the following command to go to the client installation directory:</span><p><p id="mrs_01_0473__p6252114252417"><strong id="mrs_01_0473__b4252142172412">cd /opt/hadoopclient</strong></p>
|
|
</p></li><li id="mrs_01_0473__li53331757123013"><span>Run the following command to initialize environment variables:</span><p><p id="mrs_01_0473__p933312571307"><b><span class="cmdname" id="mrs_01_0473__cmdname73338575306">source /opt/hadoopclient/bigdata_env</span></b></p>
|
|
</p></li><li id="mrs_01_0473__li33331457203014"><span>If Kerberos authentication is enabled for the cluster, perform the following steps. If not, skip this whole step.</span><p><ol type="a" id="mrs_01_0473__ol17333457103012"><li id="mrs_01_0473__li20333185716302">Prepare a user for submitting Flink jobs.</li><li id="mrs_01_0473__li833445710304">Log in to Manager and download the authentication credential.<p id="mrs_01_0473__p15647122144119"><a name="mrs_01_0473__li833445710304"></a><a name="li833445710304"></a>Log in to Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager (MRS 3.x or Later)</a>. Choose <strong id="mrs_01_0473__b8326183743918">System</strong> > <strong id="mrs_01_0473__b6326183793920">Permission</strong> > <strong id="mrs_01_0473__b1326133715394">Manage User</strong>. On the displayed page, locate the row that contains the added user, click <strong id="mrs_01_0473__b53268378391">More</strong> in the <strong id="mrs_01_0473__b123271237193918">Operation</strong> column, and select <strong id="mrs_01_0473__b232723714393">Download authentication credential</strong>.</p>
|
|
</li><li id="mrs_01_0473__li16334205720304">Decompress the downloaded authentication credential package and copy the <strong id="mrs_01_0473__b1075741809">user.keytab</strong> file to the client node, for example, to the <span class="filepath" id="mrs_01_0473__filepath153341157173013"><b>/opt/hadoopclient/Flink/flink/conf</b></span> directory on the client node. If the client is installed on a node outside the cluster, copy the <strong id="mrs_01_0473__b39268780">krb5.conf</strong> file to the <strong id="mrs_01_0473__b324222745">/etc/</strong> directory on this node.</li><li id="mrs_01_0473__li1966731417196">Append the service IP address of the node where the client is installed, floating IP address of Manager, and IP address of the master node to the <strong id="mrs_01_0473__b115620306572">jobmanager.web.access-control-allow-origin</strong> and <strong id="mrs_01_0473__b1635171102317">jobmanager.web.allow-access-address</strong> configuration item in the <strong id="mrs_01_0473__b71561530115718">/opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong> file. Use commas (,) to separate IP addresses.<pre class="screen" id="mrs_01_0473__screen161072058112210">jobmanager.web.access-control-allow-origin: <em id="mrs_01_0473__i19107155819226">xx.xx.xxx.xxx</em>,<em id="mrs_01_0473__i9258143382713">xx.xx.xxx.xxx</em>,<em id="mrs_01_0473__i839663411278">xx.xx.xxx.xxx</em>
|
|
jobmanager.web.allow-access-address: <em id="mrs_01_0473__i1710735812221">xx.xx.xxx.xxx</em>,<em id="mrs_01_0473__i15162384275">xx.xx.xxx.xxx</em>,<em id="mrs_01_0473__i137311391272">xx.xx.xxx.xxx</em></pre>
|
|
<div class="note" id="mrs_01_0473__note1344815262211"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0473__ul20448205212212"><li id="mrs_01_0473__li2448105210225">To obtain the service IP address of the node where the client is installed, perform the following operations:<ul id="mrs_01_0473__ul10448125211221"><li id="mrs_01_0473__li64481652162212">Node inside the cluster:<p id="mrs_01_0473__p04471452132210"><a name="mrs_01_0473__li64481652162212"></a><a name="li64481652162212"></a>In the navigation tree of the MRS management console, choose <span class="uicontrol" id="mrs_01_0473__uicontrol142906190331827"><b>Clusters > Active Clusters</b></span>, select a cluster, and click its name to switch to the cluster details page.</p>
|
|
<p id="mrs_01_0473__p1244825282217">On the <strong id="mrs_01_0473__b20538843133413">Nodes</strong> tab page, view the IP address of the node where the client is installed.</p>
|
|
</li><li id="mrs_01_0473__li0448175211229">Node outside the cluster: IP address of the ECS where the client is installed.</li></ul>
|
|
</li><li id="mrs_01_0473__li19448252112213">To obtain the floating IP address of Manager, perform the following operations:<ul id="mrs_01_0473__ul4448195232218"><li id="mrs_01_0473__li044835202218">In the navigation tree of the MRS management console, choose <span class="uicontrol" id="mrs_01_0473__uicontrol5707124133714"><b>Clusters > Active Clusters</b></span>, select a cluster, and click its name to switch to the cluster details page.<p id="mrs_01_0473__p44484526225">On the <strong id="mrs_01_0473__b114301229143717">Nodes</strong> tab page, view the <strong id="mrs_01_0473__b1943572953719">Name</strong>. The node that contains <strong id="mrs_01_0473__b1043542913712">master1</strong> in its name is the Master1 node. The node that contains <strong id="mrs_01_0473__b4436729173716">master2</strong> in its name is the Master2 node.</p>
|
|
</li></ul>
|
|
<ul id="mrs_01_0473__ul544835272214"><li id="mrs_01_0473__li12448155252214">Log in to the Master2 node remotely, and run the <strong id="mrs_01_0473__b23132356731827">ifconfig</strong> command. In the command output, <strong id="mrs_01_0473__b92255447131827">eth0:wsom</strong> indicates the floating IP address of MRS Manager. Record the value of <strong id="mrs_01_0473__b180334352131827">inet</strong>. If the floating IP address of MRS Manager cannot be queried on the Master2 node, switch to the Master1 node to query and record the floating IP address. If there is only one Master node, query and record the cluster manager IP address of the Master node.</li></ul>
|
|
</li></ul>
|
|
</div></div>
|
|
</li><li id="mrs_01_0473__li133495753011">Configure security authentication by adding the <strong id="mrs_01_0473__b11651629111">keytab</strong> path and username in the <strong id="mrs_01_0473__b01652021711">/opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong> configuration file.<p id="mrs_01_0473__p20334205793011"><strong id="mrs_01_0473__b1359440071">security.kerberos.login.keytab: </strong><em id="mrs_01_0473__i108123442"><user.keytab file path></em></p>
|
|
<p id="mrs_01_0473__p03351657103018"><strong id="mrs_01_0473__b509549175">security.kerberos.login.principal: </strong><em id="mrs_01_0473__i517153231"><Username></em></p>
|
|
<p id="mrs_01_0473__p15335185733015">Example:</p>
|
|
<p id="mrs_01_0473__p1133575703016">security.kerberos.login.keytab: /opt/hadoopclient/Flink/flink/conf/user.keytab</p>
|
|
<p id="mrs_01_0473__p1633515710303">security.kerberos.login.principal: test</p>
|
|
</li><li id="mrs_01_0473__li83351457113016">Generate the <strong id="mrs_01_0473__b8539956193818">generate_keystore.sh</strong> script and place it in the <strong id="mrs_01_0473__b20540156103813">bin</strong> directory of the Flink client. In the <strong id="mrs_01_0473__b455545617389">bin</strong> directory of the Flink client, run the following command to perform security hardening. For details, see <span id="mrs_01_0473__ph1656815565384"><a href="https://docs.otc.t-systems.com/cmpntguide/mrs/mrs_01_1583.html" target="_blank" rel="noopener noreferrer">Authentication and Encryption</a></span>. Set <strong id="mrs_01_0473__b19568456123816">password</strong> in the following command to a password for submitting jobs:<p id="mrs_01_0473__p13335057183010"><strong id="mrs_01_0473__b1233505716309">sh generate_keystore.sh <<em id="mrs_01_0473__i23351257103012">password</em>></strong></p>
|
|
<p id="mrs_01_0473__p13335205753018">The script automatically replaces the SSL value in the <strong id="mrs_01_0473__b6645247134117">/opt/hadoopclient/Flink/flink/conf/flink-conf.yaml</strong> file.</p>
|
|
<p id="mrs_01_0473__p1936162285618"><strong id="mrs_01_0473__b23611622105611">sh generate_keystore.sh <<em id="mrs_01_0473__i736116225562">password</em>></strong></p>
|
|
<div class="note" id="mrs_01_0473__note33361357113017"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><div class="p" id="mrs_01_0473__p922342252312">After authentication and encryption, the <span class="filepath" id="mrs_01_0473__filepath611046175216"><b>flink.keystore</b></span> and <span class="filepath" id="mrs_01_0473__filepath2110269526"><b>flink.truststore</b></span> files are generated in the <strong id="mrs_01_0473__b911146155211">conf</strong> directory on the Flink client and the following configuration items are set to the default values in the <strong id="mrs_01_0473__b108937913534">flink-conf.yaml</strong> file:<ul id="mrs_01_0473__ul89714582189"><li id="mrs_01_0473__li199711758121813">Set <span class="parmname" id="mrs_01_0473__parmname14845443502945"><b>security.ssl.keystore</b></span> to the absolute path of the <span class="filepath" id="mrs_01_0473__filepath130164191002945"><b>flink.keystore</b></span> file.</li><li id="mrs_01_0473__li103211421910">Set <span class="parmname" id="mrs_01_0473__parmname113980033302945"><b>security.ssl.truststore</b></span> to the absolute path of the <span class="filepath" id="mrs_01_0473__filepath130465322902945"><b>flink.truststore</b></span> file.</li></ul>
|
|
<ul id="mrs_01_0473__ul967157151910"><li id="mrs_01_0473__li16719718197">Set <strong id="mrs_01_0473__b56252958602945">security.cookie</strong> to a random password automatically generated by the <strong id="mrs_01_0473__b134451104202945">generate_keystore.sh</strong> script.</li><li id="mrs_01_0473__li83115101193">By default, <strong id="mrs_01_0473__b48304986302945">security.ssl.encrypt.enabled</strong> is set to <strong id="mrs_01_0473__b1959420973211">false</strong> in the <strong id="mrs_01_0473__b102365698902945">flink-conf.yaml</strong> file by default. The <strong id="mrs_01_0473__b18423148902945">generate_keystore.sh</strong> script sets <strong id="mrs_01_0473__b65165231502945">security.ssl.key-password</strong>, <strong id="mrs_01_0473__b181792140702945">security.ssl.keystore-password</strong>, and <strong id="mrs_01_0473__b133977440502945">security.ssl.truststore-password</strong> to the password entered when the <strong id="mrs_01_0473__b48245844902945">generate_keystore.sh</strong> script is called.</li></ul>
|
|
<ul id="mrs_01_0473__ul98721361913"><li id="mrs_01_0473__li987171319191">For MRS 3.<em id="mrs_01_0473__i102612524489">x</em> or later, if ciphertext is required and <strong id="mrs_01_0473__b426205224819">security.ssl.encrypt.enabled</strong> is set to <strong id="mrs_01_0473__b02625294818">true</strong> in the <strong id="mrs_01_0473__b827115213486">flink-conf.yaml</strong> file, the <strong id="mrs_01_0473__b22720523483">generate_keystore.sh</strong> script does not set <strong id="mrs_01_0473__b112745264810">security.ssl.key-password</strong>, <strong id="mrs_01_0473__b52714528486">security.ssl.keystore-password</strong>, and <strong id="mrs_01_0473__b2271052154816">security.ssl.truststore-password</strong>. To obtain the values, use the Manager plaintext encryption API by running <strong id="mrs_01_0473__b1327175213489">curl -k -i -u </strong><em id="mrs_01_0473__i122875294815">Username</em><strong id="mrs_01_0473__b15281520486">:</strong><em id="mrs_01_0473__i11289527483">Password</em><strong id="mrs_01_0473__b0289528482"> -X POST -HContent-type:application/json -d '{"plainText":"</strong><em id="mrs_01_0473__i928175219481">Password</em><strong id="mrs_01_0473__b12281952194818">"}' 'https://</strong><em id="mrs_01_0473__i1728752194820">x.x.x.x</em><strong id="mrs_01_0473__b112985212487">:28443/web/api/v2/tools/encrypt'</strong>.<p id="mrs_01_0473__p9490344151413">In the preceding command, <em id="mrs_01_0473__i19443430602945">Username</em><strong id="mrs_01_0473__b28958321702945">:</strong><em id="mrs_01_0473__i169507529002945">Password</em> indicates the user name and password for logging in to the system. The password of <strong id="mrs_01_0473__b138077002702945">"plainText"</strong> indicates the one used to call the <strong id="mrs_01_0473__b187584138102945">generate_keystore.sh</strong> script. <em id="mrs_01_0473__i180426901302945">x.x.x.x</em> indicates the floating IP address of Manager.</p>
|
|
</li></ul>
|
|
</div>
|
|
</div></div>
|
|
</li><li id="mrs_01_0473__li1633718578306">Configure paths for the client to access the <strong id="mrs_01_0473__b19625050">flink.keystore</strong> and <strong id="mrs_01_0473__b1536252530">flink.truststore</strong> files.<ul id="mrs_01_0473__ul9337115713309"><li id="mrs_01_0473__li133755793018">Absolute path: After the script is executed, the file path of <strong id="mrs_01_0473__b944820122120">flink.keystore</strong> and <strong id="mrs_01_0473__b8454812814">flink.truststore</strong> is automatically set to the absolute path <strong id="mrs_01_0473__b14454181212114">/opt/hadoopclient/Flink/flink/conf/</strong> in the <strong id="mrs_01_0473__b1945419121116">flink-conf.yaml</strong> file. In this case, you need to move the <strong id="mrs_01_0473__b14454612111">flink.keystore</strong> and <strong id="mrs_01_0473__b645411126115">flink.truststore</strong> files from the <strong id="mrs_01_0473__b445519128115">conf</strong> directory to this absolute path on the Flink client and Yarn nodes.</li><li id="mrs_01_0473__li1833712578307">Relative path: Perform the following steps to set the file path of <strong id="mrs_01_0473__b982394273">flink.keystore</strong> and <strong id="mrs_01_0473__b1207022704">flink.truststore</strong> to the relative path and ensure that the directory where the Flink client command is executed can directly access the relative paths.<ol class="substepthirdol" id="mrs_01_0473__ol433715570300"><li id="mrs_01_0473__li19337195716308">Create a directory, for example, <strong id="mrs_01_0473__b1873515204116">ssl</strong>, in <strong id="mrs_01_0473__b1574116204110">/opt/hadoopclient/Flink/flink/conf/</strong>.<p id="mrs_01_0473__p113733811110"><strong id="mrs_01_0473__b1537938614">cd /opt/hadoopclient/Flink/flink/conf/</strong></p>
|
|
<p id="mrs_01_0473__p2377386114"><strong id="mrs_01_0473__b1138133814117">mkdir ssl</strong></p>
|
|
</li><li id="mrs_01_0473__li1333725710307">Move the <strong id="mrs_01_0473__b43521528816">flink.keystore</strong> and <strong id="mrs_01_0473__b1835817281718">flink.truststore</strong> files to the <strong id="mrs_01_0473__b8358428813">/opt/hadoopclient/Flink/flink/conf/ssl/</strong> directory.<p id="mrs_01_0473__p915085716118"><strong id="mrs_01_0473__b191504571112">mv flink.keystore ssl/</strong></p>
|
|
<p id="mrs_01_0473__p1150185714118"><strong id="mrs_01_0473__b1315025715114">mv flink.truststore ssl/</strong></p>
|
|
</li><li id="mrs_01_0473__li7337195753011">Change the values of the following parameters to relative paths in the <strong id="mrs_01_0473__b39614149425">flink-conf.yaml</strong> file:<pre class="screen" id="mrs_01_0473__screen333775712308">security.ssl.keystore: ssl/flink.keystore
|
|
security.ssl.truststore: ssl/flink.truststore</pre>
|
|
</li></ol>
|
|
</li></ul>
|
|
</li></ol>
|
|
</p></li><li id="mrs_01_0473__li933835753012"><span>Run a wordcount job.</span><p><div class="notice" id="mrs_01_0473__note633817574302"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="mrs_01_0473__p14338157153018">To submit or run jobs on Flink, the user must have the following permissions:</p>
|
|
<ul id="mrs_01_0473__ul11338155715300"><li id="mrs_01_0473__li7338357193020">If Ranger authentication is enabled, the current user must belong to the <strong id="mrs_01_0473__b480438346">hadoop</strong> group or the user has been granted the <strong id="mrs_01_0473__b896587505">/flink</strong> read and write permissions in Ranger.</li><li id="mrs_01_0473__li163389576308">If Ranger authentication is disabled, the current user must belong to the <strong id="mrs_01_0473__b108133453">hadoop</strong> group.</li></ul>
|
|
</div></div>
|
|
<ul id="mrs_01_0473__ul15338205763016"><li id="mrs_01_0473__li733811570308">Normal cluster (Kerberos authentication disabled)<ul id="mrs_01_0473__ul9338105723018"><li id="mrs_01_0473__li8338757133020">Run the following commands to start a session and submit a job in the session:<p id="mrs_01_0473__p1342823442319"><a name="mrs_01_0473__li8338757133020"></a><a name="li8338757133020"></a><strong id="mrs_01_0473__b17975193511241">yarn-session.sh -nm "</strong><em id="mrs_01_0473__i13993173514241">session-name</em><strong id="mrs_01_0473__b5976143517240">"</strong></p>
|
|
<p id="mrs_01_0473__p13428934122320"><strong id="mrs_01_0473__b1428163472317">flink run /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li><li id="mrs_01_0473__li533815718304">Run the following command to submit a single job on Yarn:<p id="mrs_01_0473__p1473934162314"><a name="mrs_01_0473__li533815718304"></a><a name="li533815718304"></a><strong id="mrs_01_0473__b6739194152312">flink run -m yarn-cluster /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li></ul>
|
|
</li><li id="mrs_01_0473__li163391557193019">Security cluster (Kerberos authentication enabled)<ul id="mrs_01_0473__ul1833985716309"><li id="mrs_01_0473__li19339115715308">If the <strong id="mrs_01_0473__b385019017">flink.keystore</strong> and <strong id="mrs_01_0473__b1217591367">flink.truststore</strong> files are stored in the absolute path:<ul id="mrs_01_0473__ul173396576303"><li id="mrs_01_0473__li19339657143012">Run the following commands to start a session and submit a job in the session:<p id="mrs_01_0473__p6964104862317"><a name="mrs_01_0473__li19339657143012"></a><a name="li19339657143012"></a><strong id="mrs_01_0473__b14917122416249">yarn-session.sh -nm "</strong><em id="mrs_01_0473__i2091920241249">session-name</em><strong id="mrs_01_0473__b9917112413242">"</strong></p>
|
|
<p id="mrs_01_0473__p0964184818235"><strong id="mrs_01_0473__b596474813233">flink run /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li><li id="mrs_01_0473__li733916578303">Run the following command to submit a single job on Yarn:<p id="mrs_01_0473__p5681185620234"><a name="mrs_01_0473__li733916578303"></a><a name="li733916578303"></a><strong id="mrs_01_0473__b18681115610239">flink run -m yarn-cluster /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li></ul>
|
|
</li><li id="mrs_01_0473__li5339105712304">If the <strong id="mrs_01_0473__b1991377813">flink.keystore</strong> and <strong id="mrs_01_0473__b294345740">flink.truststore</strong> file are stored in the relative path:<ul id="mrs_01_0473__ul534018571301"><li id="mrs_01_0473__li2034055717309">In the same directory of SSL, run the following commands to start a session and submit jobs in the session. The SSL directory is a relative path. For example, if the SSL directory is <strong id="mrs_01_0473__b134475451618">opt/hadoopclient/Flink/flink/conf/</strong>, then run the following commands in this directory:<p id="mrs_01_0473__p139937212243"><strong id="mrs_01_0473__b17328201715243">yarn-session.sh -t ssl/ -nm "</strong><em id="mrs_01_0473__i10330151710246">session-name</em><strong id="mrs_01_0473__b632861710249">"</strong></p>
|
|
<p id="mrs_01_0473__p10993422248"><strong id="mrs_01_0473__b18993112142413">flink run /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li><li id="mrs_01_0473__li9340857183018">Run the following command to submit a single job on Yarn:<p id="mrs_01_0473__p12536121016249"><a name="mrs_01_0473__li9340857183018"></a><a name="li9340857183018"></a><strong id="mrs_01_0473__b14536410202414">flink run -m yarn-cluster -yt ssl/ /opt/hadoopclient/Flink/flink/examples/streaming/WordCount.jar</strong></p>
|
|
</li></ul>
|
|
</li></ul>
|
|
</li></ul>
|
|
</p></li><li id="mrs_01_0473__li1340195713300"><span>After the job has been successfully submitted, the following information is displayed on the client:</span><p><div class="fignone" id="mrs_01_0473__fig15340185743012"><span class="figcap"><b>Figure 5 </b>Job submitted successfully on Yarn</span><br><span><img id="mrs_01_0473__image153406577305" src="en-us_image_0000001349090457.png"></span></div>
|
|
<div class="fignone" id="mrs_01_0473__fig203401757133017"><span class="figcap"><b>Figure 6 </b>Session started successfully</span><br><span><img id="mrs_01_0473__image1234055711304" src="en-us_image_0000001349170353.png"></span></div>
|
|
<div class="fignone" id="mrs_01_0473__fig03403579301"><span class="figcap"><b>Figure 7 </b>Job submitted successfully in the session</span><br><span><img id="mrs_01_0473__image1934075793011" src="en-us_image_0000001348770649.png"></span></div>
|
|
</p></li><li id="mrs_01_0473__li1334075703011"><span>Go to the native YARN service page, find the application of the job, and click the application name to go to the job details page. For details, see <span id="mrs_01_0473__ph193408576309"><a href="https://docs.otc.t-systems.com/cmpntguide/mrs/mrs_01_0784.html" target="_blank" rel="noopener noreferrer">Viewing Flink Job Information</a><span id="mrs_01_0473__ph12479827195012"></span></span>.</span><p><ul id="mrs_01_0473__ul234118572307"><li id="mrs_01_0473__li14341157183013">If the job is not completed, click <strong id="mrs_01_0473__b52268204">Tracking URL</strong> to go to the native Flink page and view the job running information.</li><li id="mrs_01_0473__li1934155773015">If the job submitted in a session has been completed, you can click <strong id="mrs_01_0473__b1046313317214">Tracking URL</strong> to log in to the native Flink service page to view job information.<div class="fignone" id="mrs_01_0473__fig12341157133015"><span class="figcap"><b>Figure 8 </b>Application</span><br><span><img id="mrs_01_0473__image63529265287" src="en-us_image_0000001438951649.png"></span></div>
|
|
</li></ul>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0591.html">Using Flink</a></div>
|
|
</div>
|
|
</div>
|
|
|