forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
42 lines
5.6 KiB
HTML
42 lines
5.6 KiB
HTML
<a name="mrs_01_2057"></a><a name="mrs_01_2057"></a>
|
|
|
|
<h1 class="topictitle1">What Can I Do If an Error Occurs when I Access the Application Page Because the Application Cached by HistoryServer Is Recycled?</h1>
|
|
<div id="body1595920225549"><div class="section" id="mrs_01_2057__s52854ce0eb01400897874068eb3ceac6"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2057__a6f278778020b4bfa8612cd62f51f6d31">An error occurs when I access a Spark application page on the HistoryServer page.</p>
|
|
<p id="mrs_01_2057__a87dd778864c942ec8fb8c4b1bda38afb">Check the HistoryServer logs. The "FileNotFound" exception is found. The related logs are as follows:</p>
|
|
<pre class="screen" id="mrs_01_2057__sb5de590deaeb4fb6b369448e3240e40b">2016-11-22 23:58:03,694 | WARN | [qtp55429210-232] | /history/application_1479662594976_0001/stages/stage/ | org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:628)
|
|
java.io.FileNotFoundException: ${BIGDATA_HOME}/tmp/spark/jobHistoryTemp/blockmgr-5f1f6aca-2303-4290-9845-88fa94d78480/09/temp_shuffle_11f82aaf-e226-46dc-b1f0-002751557694 (No such file or directory)</pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_2057__sc60702f4a723434ab94798b2974aa07f"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_2057__a91f0482aa22b46e884fa1e31d62cf147">If a Spark application with a large number of tasks is run on the HistoryServer page, the memory overflows to disk and files with the <strong id="mrs_01_2057__b18326639609341">temp_shuffle</strong> prefix are generated. </p>
|
|
<p id="mrs_01_2057__a83d4c4538c974e25b72ff3573e72ffb7">By default, HistoryServer caches 50 Spark applications (determined by the <span class="parmname" id="mrs_01_2057__parmname9419905959341"><b>spark.history.retainedApplications</b></span> configuration item). When the number of Spark applications in the memory exceeds 50, HistoryServer reclaims the first cached Spark application and clears the corresponding <strong id="mrs_01_2057__b4584676829341">temp_shuffle</strong> file.</p>
|
|
<p id="mrs_01_2057__a21220dab57f548038d68182b1d245b57">When a user is viewing Spark applications to be recycled, the <strong id="mrs_01_2057__b12639935429341">temp_shuffle</strong> file may not be found. As a result, the current page cannot be accessed.</p>
|
|
<p id="mrs_01_2057__ae4ebe7bba0244ec59ed9b583a037a446">If the preceding problem occurs, use either of the following methods to solve the problem:</p>
|
|
<ul id="mrs_01_2057__uc420aa4d6f0e46c496907fd7e07c600d"><li id="mrs_01_2057__l67b7b53dc8584fdfa7a68ed2496c1df8">Access the HistoryServer page of the Spark application again. The correct page information is displayed.</li><li id="mrs_01_2057__l1fa676394a3e475cae7823c767bad68d">If more than 50 Spark applications need to be accessed at the same time, increase the value of <span class="parmname" id="mrs_01_2057__pfc92ac5c54b94883a8511ac157d0f8ab"><b>spark.history.retainedApplications</b></span>.<div class="p" id="mrs_01_2057__a7d7e82de84674777a59b76c72c046e99">Log in to FusionInsight Manager, choose<span id="mrs_01_2057__text664014226348"> <strong id="mrs_01_2057__b1063942213342">Cluster</strong> > <em id="mrs_01_2057__i186402224345">Name of the desired cluster</em></span> > <strong id="mrs_01_2057__b17640922103412">Service</strong> > <strong id="mrs_01_2057__b2064012223414">Spark2x</strong> > <strong id="mrs_01_2057__b146401022193415">Configuration</strong>, and click <strong id="mrs_01_2057__b964142293418">All Configurations</strong>. In the navigation tree on the left, choose <strong id="mrs_01_2057__b1664122253419">JobHistory</strong><strong id="mrs_01_2057__b196411922113414">2x</strong> > <strong id="mrs_01_2057__b1164117224346">GUI</strong>, and set parameters.
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_2057__t8fe40c6892824846bd688bbf1d1a3812" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_2057__r85f1672165d849f1960d45c04034269f"><th align="left" class="cellrowborder" valign="top" width="19.3%" id="mcps1.3.2.6.2.2.9.2.4.1.1"><p id="mrs_01_2057__a3ed21e7d07944921969c27a05653df21"><strong id="mrs_01_2057__af9be82970f3e4ee19b441c6a8a454fe1">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="65.91%" id="mcps1.3.2.6.2.2.9.2.4.1.2"><p id="mrs_01_2057__a47dbea2a0c7b45069468c67a406ca73b"><strong id="mrs_01_2057__a1bc7a6319ae8458ea5c745d352c0ab34">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="14.790000000000001%" id="mcps1.3.2.6.2.2.9.2.4.1.3"><p id="mrs_01_2057__a19fa0646cf35486cb0c5b47c603d2571"><strong id="mrs_01_2057__a64686f5f00114985ac401266cc8e6b12">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_2057__r845e04c957234bee9fdda54fdf5e90d0"><td class="cellrowborder" valign="top" width="19.3%" headers="mcps1.3.2.6.2.2.9.2.4.1.1 "><p id="mrs_01_2057__abb458eafe87b42a590d81051a18aecaf">spark.history.retainedApplications</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.91%" headers="mcps1.3.2.6.2.2.9.2.4.1.2 "><p id="mrs_01_2057__ad272d181b8834c0b847b5796faf295e1">Number of Spark applications cached by HistoryServer. When the number of applications to be cached exceeds the value of this parameter, HistoryServer reclaims the first cached Spark application.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.790000000000001%" headers="mcps1.3.2.6.2.2.9.2.4.1.3 "><p id="mrs_01_2057__a9de98922aa344074a2aadf371b7761fc">50</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2002.html">Common Issues About Spark2x</a></div>
|
|
</div>
|
|
</div>
|
|
|