forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
47 lines
4.6 KiB
HTML
47 lines
4.6 KiB
HTML
<a name="mrs_01_1958"></a><a name="mrs_01_1958"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Whether Spark Obtains HBase Tokens</h1>
|
|
<div id="body1595920207028"><div class="section" id="mrs_01_1958__sa62e39c2cdd74d74902eb36a631a41d8"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1958__ae07149ee4e4c443bacaa11b61e6f8138">When Spark is used to submit tasks, the driver obtains tokens from HBase by default. To access HBase, you need to configure the <strong id="mrs_01_1958__b65651951183">jaas.conf</strong> file for security authentication. If the <strong id="mrs_01_1958__b1276754619910">jaas.conf</strong> file is not configured, the application will fail to run.</p>
|
|
<p id="mrs_01_1958__a551f9576f1f840b9abd91e6302a24963">Therefore, perform the following operations based on whether the application involves HBase:</p>
|
|
<ul id="mrs_01_1958__u2543ceff53274c3daa4d16dd75216f96"><li id="mrs_01_1958__ld93f94a4f4ab4b10ab1ed54f63dbead1">If the application does not involve HBase, you do not need to obtain the HBase tokens. In this case, set <strong id="mrs_01_1958__b1444013582919">spark.yarn.security.credentials.hbase.enabled</strong> to <strong id="mrs_01_1958__b0445185812915">false</strong>.</li><li id="mrs_01_1958__l01c44487c2fc4125a864b6db94c0ff68">If the application involves HBase, set <strong id="mrs_01_1958__b13535193151017">spark.yarn.security.credentials.hbase.enabled</strong> to <strong id="mrs_01_1958__b154013310107">true</strong> and configure the <strong id="mrs_01_1958__b1754023121012">jaas.conf</strong> file on the driver as follows:<pre class="screen" id="mrs_01_1958__s2e598db6b8604c2e8fad8190070b433c">{client}/spark/bin/spark-sql --master yarn-client --principal {principal} --keytab {keytab} --driver-java-options "-Djava.security.auth.login.config={LocalPath}/jaas.conf"</pre>
|
|
<p id="mrs_01_1958__a4d00f8d9c7a34b3ca15e97ad87248b2a">Specify Keytab and Principal in the <strong id="mrs_01_1958__b8567171771010">jaas.conf</strong> file. The following is an example:</p>
|
|
<pre class="screen" id="mrs_01_1958__sd8f6bf95704b413a81eba215c089e4e4">Client {
|
|
com.sun.security.auth.module.Krb5LoginModule required
|
|
useKeyTab=true
|
|
keyTab = "{LocalPath}/user.keytab"
|
|
principal="<strong id="mrs_01_1958__b2020218255101">super@</strong><em id="mrs_01_1958__i15206142519105"><System domain name></em>"
|
|
useTicketCache=false
|
|
debug=false;
|
|
};</pre>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1958__s4207df30f119448c8b958d89b40286ae"><h4 class="sectiontitle">Configuration</h4><p id="mrs_01_1958__ac52fb826929e4271b83e42e20e7af23c">Configure the following parameter in the <strong id="mrs_01_1958__b277883011017">spark-defaults.conf</strong> file of the Spark client.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1958__t6e01e2e5a19b486caadef2ac495f1817" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1958__r0f0f70a3b53f4462a4d2daf14e57f89a"><th align="left" class="cellrowborder" valign="top" width="32.800000000000004%" id="mcps1.3.2.3.2.4.1.1"><p id="mrs_01_1958__a7ea98270112a48fb88043da3b53e59ab">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="47.25%" id="mcps1.3.2.3.2.4.1.2"><p id="mrs_01_1958__a46edfb0102474ab390481707fa084b67">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="19.950000000000003%" id="mcps1.3.2.3.2.4.1.3"><p id="mrs_01_1958__ad23cf557cdb84cc78d7276a90ca73cc2">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1958__rc077016c67a64f5ea2be829f78f7effa"><td class="cellrowborder" valign="top" width="32.800000000000004%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1958__a43ead0bc0c634592b663ef5a564393f0">spark.yarn.security.credentials.hbase.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="47.25%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1958__a5b2f768cb5d4485fada5db5778475b3b">Indicates whether HBase obtains a token.</p>
|
|
<ul id="mrs_01_1958__u13db11d7f1124d8aa1685a1d98e4854d"><li id="mrs_01_1958__l93d4ca17d7924b9d97f212b409751738"><strong id="mrs_01_1958__b42496111114">true</strong>: HBase obtains a token.</li><li id="mrs_01_1958__l1d567611a91d4325ada3be78f407e97d"><strong id="mrs_01_1958__b14361171318112">false</strong>: HBase does not obtain a token.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.950000000000003%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1958__ae2df2df7000b4421bbc08773ec08ee45">false</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|