vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide/mrs_01_1930.html
Yang, Tong 6182f91ba8 MRS component operation guide_normal 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-09 14:55:21 +00:00

389 lines
42 KiB
HTML
Raw Blame History

<a name="mrs_01_1930"></a><a name="mrs_01_1930"></a>
<h1 class="topictitle1">Configuring Parameters Rapidly</h1>
<div id="body1595920205305"><div class="section" id="mrs_01_1930__s1bca7201540043519c4e21e7492cb176"><h4 class="sectiontitle">Overview</h4><p id="mrs_01_1930__a3fa46b0319b7498e912074d3575f1687">This section describes how to quickly configure common parameters and lists parameters that are not recommended to be modified when Spark2x is used.</p>
</div>
<div class="section" id="mrs_01_1930__s3479bc71fed24a4687b6d183cd9a08d9"><h4 class="sectiontitle">Common parameters to be configured</h4><p id="mrs_01_1930__a1cb5890d9ef54a71a52ca93f79434783">Some parameters have been adapted during cluster installation. However, the following parameters need to be adjusted based on application scenarios. Unless otherwise specified, the following parameters are configured in the <span class="filepath" id="mrs_01_1930__f73b3280b8c614123846bf1e0f3302730"><b>spark-defaults.conf</b></span> file on the Spark2x client.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1930__t5eaf9e2ba86b412bb9f72bc4b56f1c79" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Common parameters to be configured</caption><thead align="left"><tr id="mrs_01_1930__r2db5369137234e1d9ccc5ae5acc77d68"><th align="left" class="cellrowborder" valign="top" width="31.490000000000002%" id="mcps1.3.2.3.2.4.1.1"><p id="mrs_01_1930__a34be422fbae04c91a51a341e6d25ef58">Configuration Item</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50.870000000000005%" id="mcps1.3.2.3.2.4.1.2"><p id="mrs_01_1930__a7660856ab92d4eed88803c9e75651525">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="17.64%" id="mcps1.3.2.3.2.4.1.3"><p id="mrs_01_1930__a1e3b09da3fcb4e9188e51e2f388c4fe7">Default Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1930__r4f3ccfe7b0c94a8598598b4bb5b961de"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__a278ba8ed1bfc4c7d87c845c8553d4ad5">spark.sql.parquet.compression.codec</p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__a4cb7ae0ba0e44c30bc2a06992720d7a2">Used to set the compression format of a non-partitioned Parquet table.</p>
<p id="mrs_01_1930__ae1f0d042e6eb42799a6a63cc532e8f68">Set the queue in the <strong id="mrs_01_1930__b1184330409112239">spark-defaults.conf</strong> configuration file on the JDBCServer server.</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__ad2d476b82d48495f99b1cc9b2784a7b5">snappy</p>
</td>
</tr>
<tr id="mrs_01_1930__r0b00ef5d6241429fb3cdef4ef4df670e"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__ac73b30824cb24a83bc3c2291fa351c7d"><span id="mrs_01_1930__pd1fcb133c6ec48278e718c45f46918d7">spark.dynamicAllocation.enabled</span></p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__ab6b6bfcfe3674bf2b3608a357c4a22ab">Indicates whether to use dynamic resource scheduling, which is used to adjust the number of executors registered with the application according to scale. Currently, this parameter is valid only in Yarn mode.</p>
<p id="mrs_01_1930__a21f1da97c0e94a9bbfda46a20c25dcff">The default value for JDBCServer is <strong id="mrs_01_1930__b55600536112239">true</strong>, and that for the client is <strong id="mrs_01_1930__b1404918913112239">false</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__ab09888b8402e4bff8361da99789f9d8d"><span id="mrs_01_1930__pe824bc71402b4508a6f548368568c658">false</span></p>
</td>
</tr>
<tr id="mrs_01_1930__rf9a0f0ceeeb9448496e47013219eb164"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__aa8d92ae7393842f48e620fe6f52f3190">spark.executor.memory</p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__a7c3d6ab068594b279deebe3dd3d01eb3">Indicates the memory size used by each executor process. Its character sting is in the same format as the JVM memory (example: 512 MB or 2 GB).</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__a553bd06896d4491296d6c7adc3fb8abe">4G</p>
</td>
</tr>
<tr id="mrs_01_1930__r951da10e01bb4c9f9232b957a237d1ac"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__a300263944db245aea5d1147a7272b9cc">spark.sql.autoBroadcastJoinThreshold</p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__a74cd5ff2c1434c2a91708fd970e44eb4">Indicates the maximum value for the broadcast configuration when two tables are joined.</p>
<ul id="mrs_01_1930__ue9d6dc046bb14580af6c82a04b8e419e"><li id="mrs_01_1930__ldf8ae2e3265f40699b207422d350075d">When the size of a field in a table involved in an SQL statement is less than the value of this parameter, the system broadcasts the SQL statement.</li><li id="mrs_01_1930__laaafe1abad3a45269f99bca00b01a4ce">If the value is set to <strong id="mrs_01_1930__b697341859112239">-1</strong>, broadcast is not performed.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__a31ceed85980d4a968ac8d1d656501431">10485760</p>
</td>
</tr>
<tr id="mrs_01_1930__r3d8aa82711b849109658659d7bc1b70b"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__a148b8332f8f9471fa1c8dca4d09ba773"><span id="mrs_01_1930__p87b3a4616f2047eba7cf62633a6c6fc3">spark.yarn.queue</span></p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__a55e3efde4d694fb0beb9e5637270c244">Specifies <span id="mrs_01_1930__p6481b332111b443b8e82de90f852c850">the Yarn queue</span> where JDBCServer resides.</p>
<p id="mrs_01_1930__a1efd57ea472e486f89579b67ec08d4aa">Set the queue in the <strong id="mrs_01_1930__b2040975790112239">spark-defaults.conf</strong> configuration file on the JDBCServer server.</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__ac80ced1100a34c9fa95fd30ba3e7565a"><span id="mrs_01_1930__p821cad6fe4e749ad858e146665b43056">default</span></p>
</td>
</tr>
<tr id="mrs_01_1930__rfe95a5cebde04ababbab91e3c7b2b8fd"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__ab7f679314bab4b7090937da553031fa6">spark.driver.memory</p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__a0008c4858258430b9b0441deeb305e1c">In a large cluster, you are advised to configure the memory used by the 32 GB to 64 GB driver process, that is, the SparkContext initialization process (for example, 512 MB and 2 GB).</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__af8d8866c6a2d4f9685bf381a99647767">4G</p>
</td>
</tr>
<tr id="mrs_01_1930__red641d209d464b40b1e4d395f9432584"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__a1d550783c5114c41993cf69097b8b041"><span id="mrs_01_1930__pe380baee9d564eabbdbc617cc0c0bf0e">spark.yarn.security.credentials.hbase.enabled</span></p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__aa681f2d883a648a2bc49b8886017de43"><span id="mrs_01_1930__ph1301676912112239">Indicates whether to enable the function of obtaining HBase tokens. If the Spark on HBase function is required and a security cluster is configured, set this parameter to <strong id="mrs_01_1930__b115281447112239">true</strong>. Otherwise, set this parameter to <strong id="mrs_01_1930__b643748068112239">false</strong>.</span></p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__a90693b8c05ef4382bd52f922a827a259"><span id="mrs_01_1930__p5c3ec65105054525a27d621245654bb3">false</span></p>
</td>
</tr>
<tr id="mrs_01_1930__r8bafd0b9b3ce43e893a0fd8c41b5ccee"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__a180dcdc8d9c34d41b3c03409e83f1936"><span id="mrs_01_1930__p4694d194215d46919166a24b7795b701">spark.serializer</span></p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__a5ca06e0d72b24c9cafcc5e6de92f9c32">Used to serialize the objects that are sent over the network or need to be cached.</p>
<p id="mrs_01_1930__a7bd09a1b8507440296ecd438330f049e">The default value of Java serialization applies to any Serializable Java object, but the running speed is slow. Therefore, you are advised to use <strong id="mrs_01_1930__b716480569112239">org.apache.spark.serializer.KryoSerializer</strong> and configure Kryo serialization. It can be any subclass of <strong id="mrs_01_1930__b1028086760112239">org.apache.spark.serializer.Serializer</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__a0da8e769a6f948c884f42a4e4cd3776e"><span id="mrs_01_1930__p98cf6c5fc78944558d2430b15aecba73">org.apache.spark.serializer.JavaSerializer</span></p>
</td>
</tr>
<tr id="mrs_01_1930__r3339718da6c949b483434737544b5c43"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__a88606d445c69491caa5f37a850acdb9f">spark.executor.cores</p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__ae93482a90c9442009ff29515182e9edc">Indicates the number of kernels used by each executor.</p>
<p id="mrs_01_1930__a68461eb36c444ae1b0a4046150384cfd">Set this parameter in standalone mode and Mesos coarse-grained mode. When there are sufficient kernels, the application is allowed to execute multiple executable programs on the same worker. Otherwise, each application can run only one executable program on each worker.</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__a69c755ca3d0549ab99ddf3572678515d">1</p>
</td>
</tr>
<tr id="mrs_01_1930__r9e02057a0e6940c2894abca5dd6cd208"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__a652c5a32ddb8403b919b1291682d04d3"><span id="mrs_01_1930__p7a60a90786d94a829f89e2d2fd0454f2">spark.shuffle.service.enabled</span></p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__ad820194e20c348b28f6d67a68b322171">Indicates a long-term auxiliary service in NodeManager for improving shuffle computing performance.</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__ab92ffdaa22c24c858461823279cfb468"><span id="mrs_01_1930__p0748f597a3db453eb4fa24f308d6db09">false</span></p>
</td>
</tr>
<tr id="mrs_01_1930__rfbed8962bcae4110bb4df474421051b3"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__ae9f11209c9974315a3225a6ff240a568"><span id="mrs_01_1930__p3efbc6bd995c49e5976aae63f04f60b0">spark.sql.adaptive.enabled</span></p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__p6612254172411">Indicates whether to enable the adaptive execution framework.</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__a97f747ac2bdc46b4abef10d1bd627ce8">false</p>
</td>
</tr>
<tr id="mrs_01_1930__r13194bd174894c92bc62c4aded672de0"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__a12748c1307f94cc49e76d153836d5837">spark.executor.memoryOverhead</p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__a6769659e551744158143d316d15955df">Indicates the heap memory to be allocated to each executor, in MB.</p>
<p id="mrs_01_1930__aff64234951a2487fb4105c4801732316">This is the memory that occupies the overhead of the VM, similar to the internal string and other built-in overhead. The value increases with the executor size (usually 6% to 10%).</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__a32d53124a27a47bc93bfc37c383545b1">1 GB</p>
</td>
</tr>
<tr id="mrs_01_1930__rd65439d8b6bb43d78d9cd9a8bfbf23de"><td class="cellrowborder" valign="top" width="31.490000000000002%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1930__a260b85be95074e55908759daca2e1d7d"><span id="mrs_01_1930__p7135dcaa9af04f678848b85066804a6a">spark.streaming.kafka.direct.lifo</span></p>
</td>
<td class="cellrowborder" valign="top" width="50.870000000000005%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1930__a3d0df2b57854416d8b6538263c692239">Indicates whether to enable the LIFO function of Kafka.</p>
</td>
<td class="cellrowborder" valign="top" width="17.64%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1930__afbfc2fa2da9c4798b65ef78e0e3bd632">false</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="mrs_01_1930__s47f2107597c54548b5723bd1a3fec763"><h4 class="sectiontitle">Parameters Not Recommended to Be Modified</h4><p id="mrs_01_1930__adc00cd9846d04faea72ce319570aca31">The following parameters have been adapted during cluster installation. You are not advised to modify them.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1930__t61b2ae7216c844a9a6a6b97549e6703f" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameters not recommended to be modified</caption><thead align="left"><tr id="mrs_01_1930__r3c2bf9cd6cd94f578670f7b2b1b8f9cb"><th align="left" class="cellrowborder" valign="top" width="18.81%" id="mcps1.3.3.3.2.4.1.1"><p id="mrs_01_1930__a2722ae7b3de14d8684a55af930d26514">Configuration Item</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="26.66%" id="mcps1.3.3.3.2.4.1.2"><p id="mrs_01_1930__ad2a5525c950246aba1ca866c24372c99">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="54.53%" id="mcps1.3.3.3.2.4.1.3"><p id="mrs_01_1930__ae8d9fbd5a6d04fd7aae1131414c82e58">Default Value or Configuration Example</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1930__row5878164294617"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__p168791542184613">spark.password.factory</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__p587924212464">Selects the password parsing mode.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p387919428464">org.apache.spark.om.util.FIPasswordFactory</p>
</td>
</tr>
<tr id="mrs_01_1930__r3ad107663efd4b018e34aae1484fb53d"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a9394ba75c5474b1ab6ea8e65681e8cf8">spark.ssl.ui.protocol</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a041c2836b5554dc1a2224d490fdcd5bf">Sets the SSL protocol of the UI.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__ae11231148ee04eb5951ae5377dd332e9">TLSv1.2</p>
</td>
</tr>
<tr id="mrs_01_1930__re25c6c3f9bd647dba9e9659631e9a5ef"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__af1708f2bfd0a4e94b2f377d389fd0397">spark.yarn.archive</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a04701f807f544eb981324c10c859b0b9">Archives Spark JAR files, which are distributed to Yarn cache. If this parameter is set, the value will replace <strong id="mrs_01_1930__b1202925352112239">&lt;code&gt; spark.yarn.jars &lt;/code&gt;</strong> and be archived in the containers of all applications. The archive should contain the JAR files in its root directory. Archives can also be hosted on HDFS to speed up file distribution.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a99908d64f3184c6fb79d39bd2bfb0b88">hdfs://hacluster/user/spark2x/jars/<span id="mrs_01_1930__text13616663556">8.1.0.1</span>/spark-archive-2x.zip</p>
<div class="note" id="mrs_01_1930__note152074312620"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1930__p2520194310261"><span id="mrs_01_1930__text1456518502269">The version 8.1.0.1 is used as an example. Replace it with the actual version number.</span></p>
</div></div>
</td>
</tr>
<tr id="mrs_01_1930__r10427f4745b84551a676e810f62cc4ad"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a83b4a05c5264425b96e4ffd79167c852">spark.yarn.am.extraJavaOptions</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__afddd1349319c40e0a0334611a3c8c284">Indicates a string of extra JVM options to pass to the YARN ApplicationMaster in client mode. Use <span class="parmname" id="mrs_01_1930__p5176613728f14d56bd2b60c3c40fecbb"><b>spark.driver.extraJavaOptions</b></span> in cluster mode.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p161515286447">-Dlog4j.configuration=./__spark_conf__/__hadoop_conf__/log4j-executor.properties -Djava.security.auth.login.config=./__spark_conf__/__hadoop_conf__/jaas-zk.conf -Dzookeeper.server.principal=zookeeper/hadoop.&lt;<em id="mrs_01_1930__i108915261742">system domain name</em>&gt; -Djava.security.krb5.conf=./__spark_conf__/__hadoop_conf__/kdc.conf -Djdk.tls.ephemeralDHKeySize=2048</p>
</td>
</tr>
<tr id="mrs_01_1930__rfa1b1c935c494293bfe6612be253e68f"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a783ad9363cac4f01acfd3309b5e8a4d4">spark.shuffle.servicev2.port</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a1dbcfc8837194365b6039a4968dc67d8">Indicates the port for the shuffle service to monitor requests for obtaining data.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a5dd7ca89919848cca942375eed84d25a">27338</p>
</td>
</tr>
<tr id="mrs_01_1930__r31a74f79e6704d71a69c8c6351be522f"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__ae1302367f7cd4039a64807c61666ea49">spark.ssl.historyServer.enabled</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ae92aaf8d15e24b5e87ca4280354713ef">Sets whether the history server uses SSL.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__ad18b357446fe42639f480895c7b518eb">true</p>
</td>
</tr>
<tr id="mrs_01_1930__r24c8c4be35504db68d252a1f7f7ab04c"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a14da724bc3584fb08c34997a331cf4d6">spark.files.overwrite</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ab4c4a6f2172641f1936222f45e263e4e">When the target file exists and its content does not match that of the source file, whether to overwrite the file added through <strong id="mrs_01_1930__b7589135914481">SparkContext.addFile()</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a1c74b38408124455baab1f30eedf0833">false</p>
</td>
</tr>
<tr id="mrs_01_1930__r23af75beb8054ba7a8602dc57aa4bf55"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a867d8692d0714b9db25cb65eb7ffdc8a">spark.yarn.cluster.driver.extraClassPath</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ad074eb15ccc045e3894736a842f7f33b">Indicates the extraClassPath of the driver in Yarn-cluster mode. Set the parameter to the path and parameters of the server.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a5bfd54e189c7424eba9775bb2d17ea8e">${BIGDATA_HOME}/common/runtime/security</p>
</td>
</tr>
<tr id="mrs_01_1930__rfc977541804a4d829748d2bbc2b18d65"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__ac42634cc2dae43debad70310f3b3c07f">spark.driver.extraClassPath</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a08708b93eff14976b31a5cffd7e20c5d">Indicates the extra class path entries attached to the class path of the driver.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p41620565118">${BIGDATA_HOME}/common/runtime/security</p>
</td>
</tr>
<tr id="mrs_01_1930__r2aa604931b3845aa93c6383f7bfc3f89"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__aa4e5cade9d6341558446d1f926f3fa5a">spark.yarn.dist.innerfiles</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a2953699d6baf460dbf2c8b05ccccbe07">Sets the files that need to be uploaded to HDFS from Spark in Yarn mode.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p518615518529">/Spark_path/spark/conf/s3p.file,/Spark_path/spark/conf/locals3.jceks</p>
<p id="mrs_01_1930__afcca646efd404a8e8811e2a2ba89c4cb"><i><span class="varname" id="mrs_01_1930__v75fb65afc25b44cf8ce0ce40163c33af">Spark_path</span></i> is the installation path of the Spark client.</p>
</td>
</tr>
<tr id="mrs_01_1930__r27898b9cca7e466e89e813fec04de314"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a7bfbe80b30ef4329bc1ad0f28adc77b6">spark.sql.bigdata.register.dialect</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a1412a49bd74c4d8dabc158b7ed65dd44">Registers the SQL parser.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p1217819617589">org.apache.spark.sql.hbase.HBaseSQLParser</p>
</td>
</tr>
<tr id="mrs_01_1930__r21b3c088ac1e49d3811132ce3e8073ad"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a6b98fd64629f4a73840aeffbae57248b">spark.shuffle.manager</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ada424d5350b2436b89b38c096d64068f">Indicates the data processing mode. There are two implementation modes: sort and hash. The sort shuffle has a higher memory utilization. It is the default option in Spark 1.2 and later versions.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a5387ab264ef243bcac8cdeda071c5073">SORT</p>
</td>
</tr>
<tr id="mrs_01_1930__r276d332c2035468891d0b164f938ce04"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__ae3ef86dbfec5465f9c329603458284c4">spark.deploy.zookeeper.url</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__acf9b53f95cea41f390ca435b1975fe19">Indicates the address of ZooKeeper. Multiple addresses are separated by commas (,).</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a9c1add2953d547fb93d5af0e53ba6ed3">For example:</p>
<p id="mrs_01_1930__a7447875ba78347569149bb59d374f283">host1:2181,host2:2181,host3:2181</p>
</td>
</tr>
<tr id="mrs_01_1930__rfb0b2b522d6e43539df44ed7c74cce99"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a3172c4c5450c40769ad615a01a44f632">spark.broadcast.factory</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ae4f498ff683d4d00a97776d195cf9d54">Indicates the broadcast mode.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a784939a765e04ceabc33e3cb6beefe5e">org.apache.spark.broadcast.TorrentBroadcastFactory</p>
</td>
</tr>
<tr id="mrs_01_1930__rc04b490b783f453da8eec1c04996d410"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__p6498731194219">spark.sql.session.state.builder</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__p149373112420">Session state constructor.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p1447814319426">org.apache.spark.sql.hive.FIHiveACLSessionStateBuilder</p>
</td>
</tr>
<tr id="mrs_01_1930__r728c659942cf4827a363b5e11eb34034"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__ad5e230543a994e7aac46344acd0ea0e9">spark.executor.extraLibraryPath</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__afb06fa3055914578a7fbd0b844506f90">Sets the special library path used when the executor JVM is started.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a257e062f4a534d22b83ccb83082ed0e1">${BIGDATA_HOME}/FusionInsight_HD_<span id="mrs_01_1930__text0409219143511">8.1.0.1</span>/install/FusionInsight-Hadoop-3.1.1/hadoop/lib/native</p>
</td>
</tr>
<tr id="mrs_01_1930__r36f688574e0d4068885533a517c926cc"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a67688b2e7119421a823c79b0eb1ef4ab">spark.ui.customErrorPage</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a534d9ec6817a419881b2897d5fbf2e1b">Indicates whether to display the custom error information page when an error occurs on the page.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__abfec3b24393946789d98dd742f6cc3f3">true</p>
</td>
</tr>
<tr id="mrs_01_1930__rd55d328fc1244cf8b771ab687e26fb1d"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a31b0d0a81a964b5690170c69871d70bb">spark.httpdProxy.enable</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a14458baa449e4ba8921cd2785a9af431">Indicates whether to use the httpd proxy.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a72d8421ea848425a9b456bd077462b9d">true</p>
</td>
</tr>
<tr id="mrs_01_1930__r4d943cf6d32b4167a89121ed67728526"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__af6d26139063d4dcabe207455a8efd673">spark.ssl.ui.enabledAlgorithms</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ab326e263417240269c1a73626e6a52f4">Sets the SSL algorithm of UI.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p20570014185917">TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_DSS_WITH_AES_128_GCM_SHA256</p>
</td>
</tr>
<tr id="mrs_01_1930__r95b5667d328a4b4cafdd1e9131300abf"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a7d5b009d107e437294caea71efbd7270">spark.ui.logout.enabled</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ac30bf87866024426ab52b920e5a9d5a0">Sets the logout button for the web UI of the Spark component.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__ad16e91bb5f554e1aa10d58ee7025c5f0">true</p>
</td>
</tr>
<tr id="mrs_01_1930__rf90453bce2194ea28d298fdca171abad"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a28421131cba246c486a86c43f187a56e">spark.security.hideInfo.enabled</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a638f5251882d41c993b2fd6ee3fdb69a">Indicates whether to hide sensitive information on the UI.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a1cd441d60b49474abfec3ee0f77b082d">true</p>
</td>
</tr>
<tr id="mrs_01_1930__r8625082ae3da45aca2b444cc2f610919"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__aab99306b9b4943608f06cda9f7527a6e">spark.yarn.cluster.driver.extraLibraryPath</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ab59a196a32f34368adc52cda33b58c43">Indicates the <strong id="mrs_01_1930__b339298560112239">extraLibraryPath</strong> of the driver in Yarn-cluster mode. Set this parameter to the path and parameters of the server.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a7f01084598eb4c9889fa8f3d9337ac2f">${BIGDATA_HOME}/FusionInsight_HD_<span id="mrs_01_1930__text13456443123518">8.1.0.1</span>/install/FusionInsight-Hadoop-3.1.1/hadoop/lib/native</p>
</td>
</tr>
<tr id="mrs_01_1930__rc51cba3410d24835a45df455ca6c25f7"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__ac5fd1030570843f4811c84cee445a92b">spark.driver.extraLibraryPath</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a1fc56bba0a7441509d1e7bf9b33d8d09">Sets a special library path for starting the driver JVM.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p4621191043">${DATA_NODE_INSTALL_HOME}/hadoop/lib/native</p>
</td>
</tr>
<tr id="mrs_01_1930__re8dcab70bf2a4cfc85d655777f658d12"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a16909ef35ff841fc89cbdf74977df676">spark.ui.killEnabled</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a7284e2983d234db5a84942ca65014ed8">Allows stages and jobs to be stopped on the web UI.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a08ed5ddd503c4e0097e474859a519d5f">true</p>
</td>
</tr>
<tr id="mrs_01_1930__rebe3e5de6385483ba2b942674bf55a29"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__ab8e34762369446528158ed897bd769a7">spark.yarn.access.hadoopFileSystems</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a1bdcd4d048274ec1a27e797f4c3c5ea2">Spark can access multiple NameService instances. If there are multiple NameService instances, set this parameter to all the NameService instances and separate them with commas (,).</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a94ad1bfc748648ce847fe5cae3e48911">hdfs://hacluster,hdfs://hacluster</p>
</td>
</tr>
<tr id="mrs_01_1930__r07f0a23c5d7947ab80e1992fd656ba6d"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__ae5ef52abc54a47308e103b298a5cbc38">spark.yarn.cluster.driver.extraJavaOptions</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a931a81576e804881945c256efc7446f6">Indicates extra JVM option passed to the executor, for example, GC setting and logging. Do not set Spark attributes or heap size using this option. Instead, set Spark attributes using the SparkConf object or the <strong id="mrs_01_1930__b1809468676112239">spark-defaults.conf</strong> file specified when the spark-submit script is called. Set heap size using <strong id="mrs_01_1930__b2048391266112239">spark.executor.memory</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p13315620018">-Xloggc:&lt;LOG_DIR&gt;/gc.log -XX:+PrintGCDetails -XX:-OmitStackTraceInFastThrow -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -Dlog4j.configuration=./__spark_conf__/__hadoop_conf__/log4j-executor.properties -Djava.security.auth.login.config=./__spark_conf__/__hadoop_conf__/jaas-zk.conf -Dzookeeper.server.principal=zookeeper/hadoop.<em id="mrs_01_1930__i44810213151">&lt;system domain name&gt;</em> -Djava.security.krb5.conf=./__spark_conf__/__hadoop_conf__/kdc.conf -Djetty.version=x.y.z -Dorg.xerial.snappy.tempdir=${BIGDATA_HOME}/tmp/spark2x_app -Dcarbon.properties.filepath=./__spark_conf__/__hadoop_conf__/carbon.properties -Djdk.tls.ephemeralDHKeySize=2048</p>
</td>
</tr>
<tr id="mrs_01_1930__re3d0fa0536b94a86aa5ff4c787c7e9ae"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__acf621ab133f742549845b3394f64890d">spark.driver.extraJavaOptions</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__acbb8394426dd464c8f5001fbf9a6bb96">Indicates a series of extra JVM options passed to the driver,</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__p1385818582496">-Xloggc:${SPARK_LOG_DIR}/indexserver-omm-%p-gc.log -XX:+PrintGCDetails -XX:-OmitStackTraceInFastThrow -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:MaxDirectMemorySize=512M -XX:MaxMetaspaceSize=512M -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -XX:OnOutOfMemoryError='kill -9 %p' -Djetty.version=x.y.z -Dorg.xerial.snappy.tempdir=${BIGDATA_HOME}/tmp/spark2x/JDBCServer/snappy_tmp -Djava.io.tmpdir=${BIGDATA_HOME}/tmp/spark2x/JDBCServer/io_tmp -Dcarbon.properties.filepath=${SPARK_CONF_DIR}/carbon.properties -Djdk.tls.ephemeralDHKeySize=2048 -Dspark.ssl.keyStore=${SPARK_CONF_DIR}/child.keystore #{java_stack_prefer}</p>
</td>
</tr>
<tr id="mrs_01_1930__r17477665e1ae4d1eac314bd2f40dc664"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a04af8b3844d244df89660fcf97107bc7">spark.eventLog.overwrite</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ae2e000646abf480b8cc5a19aed7d5ca2">Indicates whether to overwrite any existing file.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a62cc7dab8fab41848fca03c7e4467c2b">false</p>
</td>
</tr>
<tr id="mrs_01_1930__r52f1440d3075496698703f777f2ae784"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a238bd28158a84f30ac228597a03cedcb">spark.eventLog.dir</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a80c4dbebbbcb4d878a6883b76c141011">Indicates the directory for logging Spark events if <strong id="mrs_01_1930__b1942529814112239">spark.eventLog.enabled</strong> is set to <strong id="mrs_01_1930__b385484385112239">true</strong>. In this directory, Spark creates a subdirectory for each application and logs events of the application in the subdirectory. You can also set a unified address similar to the HDFS directory so that the History Server can read historical files.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a42d47075bb3746f6ae58b614bf2bc290">hdfs://hacluster/spark2xJobHistory2x</p>
</td>
</tr>
<tr id="mrs_01_1930__r778b329054234323bb17ed8ed0984979"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a954b1e8ae57a48ee9a5a39ac66aea560">spark.random.port.min</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a911145e02f6a4451bcbc56b73aefe04e">Sets the minimum random port.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a9d8e8ecd8ab34eaf963fa0d724b5fcab">22600</p>
</td>
</tr>
<tr id="mrs_01_1930__r2f7b12aa09ed43669822aee98362c905"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a5cbbc6f0dbb449718f345a0a08f272ad">spark.authenticate</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ad7e8633006644f86aeb9f96fe5a1ef6e">Indicates whether Spark authenticates its internal connections. If the application is not running on Yarn, see <strong id="mrs_01_1930__b1351143444112239">spark.authenticate.secret</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__ad420b7b9e4b24bcda6a3a7cdbaf3b84d">true</p>
</td>
</tr>
<tr id="mrs_01_1930__r50968872147d419fa1ca503273360d93"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a73c9d748f5f64ac5a5df89d6a38ccd40">spark.random.port.max</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ab31573d7dac141cd942bc7fa03753bd8">Sets the maximum random port.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a69f46895a7474cd385d715d221733ba6">22899</p>
</td>
</tr>
<tr id="mrs_01_1930__rab13318abf3e4dc396db8673f8c2bc5a"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a7e8762c9314c41edb0b2bbbfe12ce230">spark.eventLog.enabled</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a3ae3b5ea778d4e57bc0cbd5f15a4233a">Indicates whether to log Spark events, which are used to reconstruct the web UI after the application execution is complete.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a9e76b4b2fbd9459b850d05929e302aea">true</p>
</td>
</tr>
<tr id="mrs_01_1930__rfc7c82a80cb64a52adb84b75ce035003"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__aa18427ffbb82481b8540bb8e0c26b397">spark.executor.extraJavaOptions</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__ad3e1ac70c9954e1693a879a46e0c6072">Indicates extra JVM option passed to the executor, for example, GC setting and logging. Do not set Spark attributes or heap size using this option.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__ac135987d786c4fe8b1bfd2b72010d2a6">-Xloggc:&lt;LOG_DIR&gt;/gc.log -XX:+PrintGCDetails -XX:-OmitStackTraceInFastThrow -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -Dlog4j.configuration=./log4j-executor.properties -Djava.security.auth.login.config=./jaas-zk.conf -Dzookeeper.server.principal=zookeeper/hadoop.&lt;<em id="mrs_01_1930__i298719288158">system domain name</em>&gt; -Djava.security.krb5.conf=./kdc.conf -Dcarbon.properties.filepath=./carbon.properties</p>
<p id="mrs_01_1930__p14818415710">-Xloggc:&lt;LOG_DIR&gt;/gc.log -XX:+PrintGCDetails -XX:-OmitStackTraceInFastThrow -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -Dlog4j.configuration=./__spark_conf__/__hadoop_conf__/log4j-executor.properties -Djava.security.auth.login.config=./__spark_conf__/__hadoop_conf__/jaas-zk.conf -Dzookeeper.server.principal=zookeeper/hadoop.<em id="mrs_01_1930__i3918131191510">&lt;system domain name&gt;</em> -Djava.security.krb5.conf=./__spark_conf__/__hadoop_conf__/kdc.conf -Dcarbon.properties.filepath=./__spark_conf__/__hadoop_conf__/carbon.properties -Djdk.tls.ephemeralDHKeySize=2048</p>
</td>
</tr>
<tr id="mrs_01_1930__rf886966d714c4cc09f7d5a405a55c9e4"><td class="cellrowborder" valign="top" width="18.81%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_1930__a5c19c960dcff452c8661a87b78130607">spark.sql.authorization.enabled</p>
</td>
<td class="cellrowborder" valign="top" width="26.66%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_1930__a852bfcee8c174cfd9ab29de0c259be25">Indicates whether to enable authentication for the Hive client.</p>
</td>
<td class="cellrowborder" valign="top" width="54.53%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_1930__a7897e5bf121445f586beb00d4897a989">true</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1928.html">Basic Operation</a></div>
</div>
</div>
View Git Blame Copy Permalink