forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
16 lines
1.7 KiB
HTML
16 lines
1.7 KiB
HTML
<a name="mrs_01_2016"></a><a name="mrs_01_2016"></a>
|
|
|
|
<h1 class="topictitle1">What Can I Do If Shuffle Fetch Fails Due to the "Timeout Waiting for Task" Exception?</h1>
|
|
<div id="body1595920219930"><div class="section" id="mrs_01_2016__s50dd42bf639b4bbfaf08e86821a27528"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2016__ae83e6342b6e947beb3652a08eb46e409">When I execute a 100 TB TPC-DS test suite in the JDBCServer mode, the "Timeout waiting for task" is displayed. As a result, shuffle fetch fails, the stage keeps retrying, and the task cannot be completed properly. What can I do?</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_2016__s06a8a9b5eefb47949cedb78ca3060ae6"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_2016__a28f4cd138eb7403798bcfefc3c3b3701">The ShuffleService function is used in JDBCServer mode. In the reduce phase, all executors obtain data from NodeManager. When the data volume reaches a level (more than 10 TB), the NodeManager may reach the bottleneck (ShuffleService is in the NodeManager process). As a result, some tasks for obtaining data time out. Therefore, the problem occurs.</p>
|
|
<p id="mrs_01_2016__a8aae314d7bda4ebeb7ecaea5c461f115">You are advised to disable ShuffleService for Spark tasks whose data volume is greater than 10 TB. That is, set <span class="parmname" id="mrs_01_2016__parmname96457309292615"><b>spark.shuffle.service.enabled</b></span> in the <span class="filepath" id="mrs_01_2016__filepath40315501592615"><b>Spark-defaults.conf</b></span> configuration file to <span class="parmvalue" id="mrs_01_2016__parmvalue2960539692615"><b>false</b></span>.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2003.html">Spark Core</a></div>
|
|
</div>
|
|
</div>
|
|
|