forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
49 lines
7.2 KiB
HTML
49 lines
7.2 KiB
HTML
<a name="mrs_01_1951"></a><a name="mrs_01_1951"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Environment Variables in Yarn-Client and Yarn-Cluster Modes</h1>
|
|
<div id="body1595920207016"><div class="section" id="mrs_01_1951__s26e829f7626a4d47acff3a322cac29ba"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1951__afd82a0e03aea4b5b867242d0d25e2dc3">Values of some configuration parameters of Spark client vary depending on its work mode (YARN-Client or YARN-Cluster). If you switch Spark client between different modes without first changing values of such configuration parameters, Spark client fails to submit jobs in the new mode.</p>
|
|
<p id="mrs_01_1951__a51519b97468f4da39510131a0b90b5b6">To avoid this, configure parameters as described in <a href="#mrs_01_1951__tbf4bf318c6154639a0cf19d9911e6f93">Table 1</a>.</p>
|
|
<ul id="mrs_01_1951__udf352af3be0d40e18bdccd6b2e0065f0"><li id="mrs_01_1951__le944f88c014d463eae2bdc266cd4fc80">In Yarn-Cluster mode, use the new parameters (path and parameters of Spark server).</li><li id="mrs_01_1951__laac50f9b6ecf41c49ae864bbf31e3c50">In Yarn-Client mode, uses the original parameters.<p id="mrs_01_1951__aab5a283e3f174a56adec1d7c575d5bad"><a name="mrs_01_1951__laac50f9b6ecf41c49ae864bbf31e3c50"></a><a name="laac50f9b6ecf41c49ae864bbf31e3c50"></a>They are <span class="parmname" id="mrs_01_1951__p54918d57da41406588b9c95a56c48d4f"><b>spark.driver.extraClassPath</b></span>, <span class="parmname" id="mrs_01_1951__p7b3ba2f6c3f942149dc297b5f6c0db47"><b>spark.driver.extraJavaOptions</b></span>, and <span class="parmname" id="mrs_01_1951__pb37da1175777499fba256376f7689dec"><b>spark.driver.extraLibraryPath</b></span>.</p>
|
|
</li></ul>
|
|
<div class="note" id="mrs_01_1951__n6f74e6f90a654c25ae760afc5dd30a15"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1951__a1efd533a04ac404f97d526aa5639390b">If you choose not to add the parameters in <a href="#mrs_01_1951__tbf4bf318c6154639a0cf19d9911e6f93">Table 1</a>, Spark client can continue to operate well in either mode but the mode switch requires changes to some of its configuration parameters.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1951__s55f4f88db6a04510b8052f873986a84f"><h4 class="sectiontitle">Configuration Parameters</h4><p id="mrs_01_1951__a7d0d698a2edf45f1a7cf1ec9c5f107ea"><strong id="mrs_01_1951__b0208105414158">Navigation path for setting parameters:</strong></p>
|
|
<p id="mrs_01_1951__adcdc3279f3cd4f12a2d9800815c849e8">On Manager, choose <span id="mrs_01_1951__text18330192912598"><strong id="mrs_01_1951__b58678452177">Cluster</strong> > <em id="mrs_01_1951__i2252142013472">Name of the desired cluster</em> > </span><strong id="mrs_01_1951__b22215701611">Services</strong> > <strong id="mrs_01_1951__b1322116717166">Spark2x</strong> > <strong id="mrs_01_1951__b1822216710164">Configurations</strong>. Click <strong id="mrs_01_1951__b1122314720166">All Configurations</strong> and enter a parameter name in the search box.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1951__tbf4bf318c6154639a0cf19d9911e6f93"></a><a name="tbf4bf318c6154639a0cf19d9911e6f93"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1951__tbf4bf318c6154639a0cf19d9911e6f93" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1951__r7dbfd24b806a4887852fd41b7128b35f"><th align="left" class="cellrowborder" valign="top" width="17.79%" id="mcps1.3.2.4.2.4.1.1"><p id="mrs_01_1951__a471cf3e6d4f34892a3dfcaac262c16cc"><strong id="mrs_01_1951__b13135231141816">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="41.48%" id="mcps1.3.2.4.2.4.1.2"><p id="mrs_01_1951__ae250bd7acf7e4e599611478214c88135"><strong id="mrs_01_1951__b1589113341819">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="40.73%" id="mcps1.3.2.4.2.4.1.3"><p id="mrs_01_1951__a7e50861023d34af2b3c01e97a6946de5"><strong id="mrs_01_1951__b387923461815">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1951__r88a52401d24540ffac87ddad1093063c"><td class="cellrowborder" valign="top" width="17.79%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_1951__a804065d541fe4d51b8695406a13e43df">spark.yarn.cluster.driver.extraClassPath</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.48%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_1951__ab76d1296fefc46baab2706678e41bdb5">Indicates the extraClassPath of the driver in Yarn-cluster mode. Set the parameter to the path and parameters of the server.</p>
|
|
<p id="mrs_01_1951__a5773aae075b043f88e2c64c84c892911">The original parameter <strong id="mrs_01_1951__b57341244101817">spark.driver.extraClassPath</strong> indicates the extraClassPath of Spark client. By using different parameters to separate the settings of Spark server from the settings of Spark client, you can switch Spark client to different modes without changing parameter values.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.73%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_1951__adc6e5e9146fd430bb7c9d18292ff44f4">${BIGDATA_HOME}/common/runtime/security</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1951__rce76867498e541f1972957d059131b9d"><td class="cellrowborder" valign="top" width="17.79%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_1951__a0a8194048cfd401a85e635a82fad515a">spark.yarn.cluster.driver.extraJavaOptions</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.48%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_1951__ada8dffd46463471fb032fbf9dc09b1c7">Indicates the extraJavaOptions of Driver in Yarn-Cluster mode and is set to path and parameters of extraJavaOptions of Spark server.</p>
|
|
<p id="mrs_01_1951__a0772d42b0dfb40ac930238fe981b362b">The original parameter <span class="parmname" id="mrs_01_1951__p51eda6a1591e405c8611a7423125563c"><b>spark.driver.extraJavaOptions</b></span> indicates the path of extraJavaOptions of Spark client. By using different parameters to separate the settings of Spark server from the settings of Spark client, you can switch Spark client to different modes without changing parameter values.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.73%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_1951__p637564731911">-Xloggc:<LOG_DIR>/indexserver-%p-gc.log -XX:+PrintGCDetails -XX:-OmitStackTraceInFastThrow -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -Dlog4j.configuration=./__spark_conf__/__hadoop_conf__/log4j-executor.properties -Dlog4j.configuration.watch=true -Djava.security.auth.login.config=./__spark_conf__/__hadoop_conf__/jaas-zk.conf -Dzookeeper.server.principal=${ZOOKEEPER_SERVER_PRINCIPAL} -Djava.security.krb5.conf=./__spark_conf__/__hadoop_conf__/kdc.conf -Djetty.version=x.y.z -Dorg.xerial.snappy.tempdir=${BIGDATA_HOME}/tmp -Dcarbon.properties.filepath=./__spark_conf__/__hadoop_conf__/carbon.properties -Djdk.tls.ephemeralDHKeySize=2048 -Dspark.ssl.keyStore=./child.keystore #{java_stack_prefer}</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|