doc-exports/docs/mrs/component-operation-guide/mrs_01_2052.html

<a name="mrs_01_2052"></a><a name="mrs_01_2052"></a>

<h1 class="topictitle1">Why Does the Spark Streaming Application Fail to Be Submitted After the Token Validity Period Expires?</h1>
<div id="body1595920224403"><div class="section" id="mrs_01_2052__sb213bc3a8c8a414fb64a61a628a7679a"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2052__a997e3a29fc314811a9c14916dd34e439">Change the validity period of the Kerberos ticket and HDFS token to 5 minutes, set <span class="parmname" id="mrs_01_2052__p4e438ecf72e7426d856182a6ac5a58c9"><b>dfs.namenode.delegation.token.renew-interval</b></span> to a value less than 60 seconds, and submit the Spark Streaming application. If the token expires, the error message below is displayed, and the application exits. Why?</p>
<pre class="screen" id="mrs_01_2052__sdff057a2efd74117b55f9447d2a34486">token (HDFS_DELEGATION_TOKEN token 17410 for spark2x) is expired</pre>
</div>
<div class="section" id="mrs_01_2052__sd5029f54ac47428d9d6494488f2ef110"><h4 class="sectiontitle">Answer</h4><ul id="mrs_01_2052__u7dd65e82c0264860a0e1dd953518a35e"><li id="mrs_01_2052__l761a24ebdf1e47798a58ce20696dd0fa">Possible causes:<p id="mrs_01_2052__aa4ac9c59e70342f9b6e6110a994c3b88"><a name="mrs_01_2052__l761a24ebdf1e47798a58ce20696dd0fa"></a><a name="l761a24ebdf1e47798a58ce20696dd0fa"></a>The credential refresh thread of the ApplicationMaster process uploads the updated credential file to the HDFS based on the<i><span class="varname" id="mrs_01_2052__varname43181303893246"> token renew period multiplied by 0.75</span></i>.</p>
<p id="mrs_01_2052__a8fe8b7958570451cbd40badef2597bd0">In the executor process, the credential refresh thread obtains the updated credential file from the HDFS based on the time ratio of the <i><span class="varname" id="mrs_01_2052__varname14511849193246">token renewal period multiplied by 0.8</span></i> to update the token in UserGroupInformation, preventing the token from being invalid.</p>
<p id="mrs_01_2052__ac9c068e10bb14d67814c04f2de56d03e">When the credential refresh thread of the executor process detects that the current time is later than the credential file update time (<i><span class="varname" id="mrs_01_2052__v0ff73ba0ab7b42e1855614bbb45c474d">token renew period x 0.8</span></i>), it waits for 1 minute and then obtains the latest credential file from the HDFS to ensure that the AM has stored the updated credential file in the HDFS.</p>
<p id="mrs_01_2052__a3a642e865e364ea28e3ed615445d1c90">When the value of <span class="parmname" id="mrs_01_2052__parmname64783026593246"><b>dfs.namenode.delegation.token.renew-interval</b></span> is less than 60 seconds, the started executor detects that the current time is later than the time when the credential file is updated. One minute later, the executor obtains the latest credential file from the HDFS. However, the token is already invalid, and the task fails to be executed. Then, other executor processes retry within 1 minute. The task also fails to run on other executors. As a result, the executors that fail to run are added to the blacklist. If no executors are available, the application exits.</p>
</li></ul>
<ul id="mrs_01_2052__u87daff84dcda47d4ae3ed4d5628b05ec"><li id="mrs_01_2052__ldb5ef5656fee4568a20861443c4ec135">Solution:<p id="mrs_01_2052__adfae6f95ee7a4eef8eb22811da56ea26"><a name="mrs_01_2052__ldb5ef5656fee4568a20861443c4ec135"></a><a name="ldb5ef5656fee4568a20861443c4ec135"></a>In the Spark application scenario, set <span class="parmname" id="mrs_01_2052__p98151dd3e1504f97abff4208d819cf0f"><b>dfs.namenode.delegation.token.renew-interval</b></span> to a value greater than 80 seconds. For details about the <span class="parmname" id="mrs_01_2052__p3755ca499b504c17b2276f392148599c"><b>dfs.namenode.delegation.token.renew-interval</b></span> parameter, see <a href="#mrs_01_2052__tccf0b141ec0540cb9da0cde471a287aa">Table 1</a>.</p>

<div class="tablenoborder"><a name="mrs_01_2052__tccf0b141ec0540cb9da0cde471a287aa"></a><a name="tccf0b141ec0540cb9da0cde471a287aa"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_2052__tccf0b141ec0540cb9da0cde471a287aa" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_2052__rbd483019a7a644b4bf00ca9925e34e58"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.3.1.2.2.4.1.1"><p id="mrs_01_2052__a02c641a81ba144f183f9cc3cc54831d7"><strong id="mrs_01_2052__a5e57877541554bb7a12a6cff7a9a4e81">Parameter</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.3.1.2.2.4.1.2"><p id="mrs_01_2052__ab5eb7f9e144b4e10a9262d08eac53429"><strong id="mrs_01_2052__af9ad9e759be04b5b94b284daa5e9fc42">Description</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.3.1.2.2.4.1.3"><p id="mrs_01_2052__a864a856523124408b7b67515be62dc1a"><strong id="mrs_01_2052__abb3172d0deb646cab8b69ef39ded6a41">Default Value</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_2052__r6ab9391ce6244aa7b633e6edc72c5c3b"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.3.1.2.2.4.1.1 "><p id="mrs_01_2052__a98f12a6d0cbc4366889f6033d52b86fd">dfs.namenode.delegation.token.renew-interval</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.3.1.2.2.4.1.2 "><p id="mrs_01_2052__a5068e62af870406bb91f942a24bbcc57">This parameter is a server parameter. It specifies the maximum lifetime to renew a token. Unit: milliseconds.</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.3.1.2.2.4.1.3 "><p id="mrs_01_2052__ac0cf5e83a4cf458f8091967e9a36cce5">86400000</p>
</td>
</tr>
</tbody>
</table>
</div>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2048.html">Spark Streaming</a></div>
</div>
</div>