forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
37 lines
3.9 KiB
HTML
37 lines
3.9 KiB
HTML
<a name="mrs_01_1980"></a><a name="mrs_01_1980"></a>
|
|
|
|
<h1 class="topictitle1">Using the external shuffle service to improve performance</h1>
|
|
<div id="body1595920217127"><div class="section" id="mrs_01_1980__section5210123615423"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1980__p197751212437">When the Spark system runs applications that contain a shuffle process, an executor process also writes shuffle data and provides shuffle data for other executors in addition to running tasks. If the executor is heavily loaded and GC is triggered, the executor cannot provide shuffle data for other executors, affecting task running.</p>
|
|
<p id="mrs_01_1980__p1777681124310">The external shuffle service is an auxiliary service in NodeManager. It captures shuffle data to reduce the load on executors. If GC occurs on an executor, tasks on other executors are not affected.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1980__section832018496431"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1980__ol64791638447"><li id="mrs_01_1980__li1348016315442"><span>Log in to FusionInsight Manager.</span></li><li id="mrs_01_1980__li174801939445"><span>Choose <span id="mrs_01_1980__text1443499075111855"><strong id="mrs_01_1980__b1834036950111855">Cluster</strong> > <em id="mrs_01_1980__i1437462877111855">Name of the desired cluster</em></span> > <strong id="mrs_01_1980__b1363480184111855">Services</strong> > <strong id="mrs_01_1980__b1462466567111855">Spark2x</strong> > <strong id="mrs_01_1980__b862505898111855">Configurations</strong>. Select <strong id="mrs_01_1980__b1830512512533">All Configurations</strong>.</span></li><li id="mrs_01_1980__li24801137443"><span>Choose <strong id="mrs_01_1980__b1807179183111855">SparkResource2x</strong> > <strong id="mrs_01_1980__b1076322959111855">Default</strong> and modify the following parameters.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1980__table19406357194515" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter list</caption><thead align="left"><tr id="mrs_01_1980__row940620577452"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.3.2.1.2.4.1.1"><p id="mrs_01_1980__p120523194612"><strong id="mrs_01_1980__b182051438465">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.3.2.1.2.4.1.2"><p id="mrs_01_1980__p3206163124613"><strong id="mrs_01_1980__b220610374618">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.3.2.1.2.4.1.3"><p id="mrs_01_1980__p4206234460"><strong id="mrs_01_1980__b1720719313463">Changed To</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1980__row1040712570458"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.3.2.1.2.4.1.1 "><p id="mrs_01_1980__p162084317465">spark.shuffle.service.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.3.2.1.2.4.1.2 "><p id="mrs_01_1980__p720913317462">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.3.2.1.2.4.1.3 "><p id="mrs_01_1980__p16210433465">true</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_1980__li1348612313444"><span>Restart the Spark2x service for the configuration to take effect.</span><p><div class="note" id="mrs_01_1980__note144870318445"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1980__p5487938443">To use the External Shuffle Service function on the Spark2x client, you need to download and install the Spark2x client again.</p>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1975.html">Spark Core Tuning</a></div>
|
|
</div>
|
|
</div>
|
|
|