forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
47 lines
5.6 KiB
HTML
47 lines
5.6 KiB
HTML
<a name="mrs_01_1958"></a><a name="mrs_01_1958"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Whether Spark Obtains HBase Tokens</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_1958__en-us_topic_0000001219230927_sa62e39c2cdd74d74902eb36a631a41d8"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1958__en-us_topic_0000001219230927_ae07149ee4e4c443bacaa11b61e6f8138">When Spark is used to submit tasks, the driver obtains tokens from HBase by default. To access HBase, you need to configure the <strong id="mrs_01_1958__en-us_topic_0000001219230927_b65651951183">jaas.conf</strong> file for security authentication. If the <strong id="mrs_01_1958__en-us_topic_0000001219230927_b1276754619910">jaas.conf</strong> file is not configured, the application will fail to run.</p>
|
|
<p id="mrs_01_1958__en-us_topic_0000001219230927_a551f9576f1f840b9abd91e6302a24963">Therefore, perform the following operations based on whether the application involves HBase:</p>
|
|
<ul id="mrs_01_1958__en-us_topic_0000001219230927_u2543ceff53274c3daa4d16dd75216f96"><li id="mrs_01_1958__en-us_topic_0000001219230927_ld93f94a4f4ab4b10ab1ed54f63dbead1">If the application does not involve HBase, you do not need to obtain the HBase tokens. In this case, set <strong id="mrs_01_1958__en-us_topic_0000001219230927_b1444013582919">spark.yarn.security.credentials.hbase.enabled</strong> to <strong id="mrs_01_1958__en-us_topic_0000001219230927_b0445185812915">false</strong>.</li><li id="mrs_01_1958__en-us_topic_0000001219230927_l01c44487c2fc4125a864b6db94c0ff68">If the application involves HBase, set <strong id="mrs_01_1958__en-us_topic_0000001219230927_b13535193151017">spark.yarn.security.credentials.hbase.enabled</strong> to <strong id="mrs_01_1958__en-us_topic_0000001219230927_b154013310107">true</strong> and configure the <strong id="mrs_01_1958__en-us_topic_0000001219230927_b1754023121012">jaas.conf</strong> file on the driver as follows:<pre class="screen" id="mrs_01_1958__en-us_topic_0000001219230927_s2e598db6b8604c2e8fad8190070b433c">{client}/spark/bin/spark-sql --master yarn-client --principal {principal} --keytab {keytab} --driver-java-options "-Djava.security.auth.login.config={LocalPath}/jaas.conf"</pre>
|
|
<p id="mrs_01_1958__en-us_topic_0000001219230927_a4d00f8d9c7a34b3ca15e97ad87248b2a">Specify Keytab and Principal in the <strong id="mrs_01_1958__en-us_topic_0000001219230927_b8567171771010">jaas.conf</strong> file. The following is an example:</p>
|
|
<pre class="screen" id="mrs_01_1958__en-us_topic_0000001219230927_sd8f6bf95704b413a81eba215c089e4e4">Client {
|
|
com.sun.security.auth.module.Krb5LoginModule required
|
|
useKeyTab=true
|
|
keyTab = "{LocalPath}/user.keytab"
|
|
principal="<strong id="mrs_01_1958__en-us_topic_0000001219230927_b2020218255101">super@</strong><em id="mrs_01_1958__en-us_topic_0000001219230927_i15206142519105"><System domain name></em>"
|
|
useTicketCache=false
|
|
debug=false;
|
|
};</pre>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1958__en-us_topic_0000001219230927_s4207df30f119448c8b958d89b40286ae"><h4 class="sectiontitle">Configuration</h4><p id="mrs_01_1958__en-us_topic_0000001219230927_ac52fb826929e4271b83e42e20e7af23c">Configure the following parameter in the <strong id="mrs_01_1958__en-us_topic_0000001219230927_b277883011017">spark-defaults.conf</strong> file of the Spark client.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1958__en-us_topic_0000001219230927_t6e01e2e5a19b486caadef2ac495f1817" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1958__en-us_topic_0000001219230927_r0f0f70a3b53f4462a4d2daf14e57f89a"><th align="left" class="cellrowborder" valign="top" width="32.800000000000004%" id="mcps1.3.2.3.2.4.1.1"><p id="mrs_01_1958__en-us_topic_0000001219230927_a7ea98270112a48fb88043da3b53e59ab">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="47.25%" id="mcps1.3.2.3.2.4.1.2"><p id="mrs_01_1958__en-us_topic_0000001219230927_a46edfb0102474ab390481707fa084b67">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="19.950000000000003%" id="mcps1.3.2.3.2.4.1.3"><p id="mrs_01_1958__en-us_topic_0000001219230927_ad23cf557cdb84cc78d7276a90ca73cc2">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1958__en-us_topic_0000001219230927_rc077016c67a64f5ea2be829f78f7effa"><td class="cellrowborder" valign="top" width="32.800000000000004%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1958__en-us_topic_0000001219230927_a43ead0bc0c634592b663ef5a564393f0">spark.yarn.security.credentials.hbase.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="47.25%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1958__en-us_topic_0000001219230927_a5b2f768cb5d4485fada5db5778475b3b">Indicates whether HBase obtains a token.</p>
|
|
<ul id="mrs_01_1958__en-us_topic_0000001219230927_u13db11d7f1124d8aa1685a1d98e4854d"><li id="mrs_01_1958__en-us_topic_0000001219230927_l93d4ca17d7924b9d97f212b409751738"><strong id="mrs_01_1958__en-us_topic_0000001219230927_b42496111114">true</strong>: HBase obtains a token.</li><li id="mrs_01_1958__en-us_topic_0000001219230927_l1d567611a91d4325ada3be78f407e97d"><strong id="mrs_01_1958__en-us_topic_0000001219230927_b14361171318112">false</strong>: HBase does not obtain a token.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1958__en-us_topic_0000001219230927_ae2df2df7000b4421bbc08773ec08ee45">false</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|