forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
80 lines
8.2 KiB
HTML
80 lines
8.2 KiB
HTML
<a name="mrs_01_1981"></a><a name="mrs_01_1981"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Dynamic Resource Scheduling in Yarn Mode</h1>
|
|
<div id="body1595920217129"><div class="section" id="mrs_01_1981__s54465d6054db4f1eb5c9843923854485"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1981__a995c003b64ab470fb45f3bfa4aaa7557">Resources are a key factor that affects Spark execution efficiency. When a long-running service (such as the JDBCServer) is allocated with multiple executors without tasks but resources of other applications are insufficient, resources are wasted and scheduled improperly.</p>
|
|
<p id="mrs_01_1981__aac2a1c6c40044dd1aa7bab78a43ea52c">Dynamic resource scheduling can add or remove executors of applications in real time based on the task load. In this way, resources are dynamically scheduled to applications.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1981__section8123144745516"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1981__o574cbfc155d84aee9e4333cb98c5f127"><li id="mrs_01_1981__l276399e445c54dae9491fd9c9d1900b7"><span>Configure the external shuffle service.</span></li><li id="mrs_01_1981__la4abc9451f944059893b4fdbfe75d533"><span>Log in to FusionInsight Manager, and choose <strong id="mrs_01_1981__b130536252493033">Cluster</strong> > <em id="mrs_01_1981__i45429156593033">Name of the desired cluster</em> > <strong id="mrs_01_1981__b20359865793033">Service</strong> > <strong id="mrs_01_1981__b49023327293033">Spark2x</strong> > <strong id="mrs_01_1981__b58374217993033">Configuration</strong> > <strong id="mrs_01_1981__b64377607993033">All Configurations</strong>. Enter <strong id="mrs_01_1981__b136161036493033">spark.dynamicAllocation.enabled</strong> in the search box and set its value to <strong id="mrs_01_1981__b67122184393033">true</strong> to enable the dynamic resource scheduling function. This function is disabled by default.</span></li></ol>
|
|
<div class="p" id="mrs_01_1981__p16776436145011"><a href="#mrs_01_1981__t55668659f37c4971bbc1bc10ed58a282">Table 1</a> lists some optional configuration items.
|
|
<div class="tablenoborder"><a name="mrs_01_1981__t55668659f37c4971bbc1bc10ed58a282"></a><a name="t55668659f37c4971bbc1bc10ed58a282"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1981__t55668659f37c4971bbc1bc10ed58a282" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters for dynamic resource scheduling</caption><thead align="left"><tr id="mrs_01_1981__re0af5e0ca79344e5b068530faaf60efd"><th align="left" class="cellrowborder" valign="top" width="29.759999999999998%" id="mcps1.3.2.3.2.2.4.1.1"><p id="mrs_01_1981__ac2f443317a34423db0e8de762a49404b"><strong id="mrs_01_1981__a4d81603a3ca546dfa177d60030f4fff3">Configuration Item</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="41.6%" id="mcps1.3.2.3.2.2.4.1.2"><p id="mrs_01_1981__a1fac7b8088e247c49eccf50822e4d9cd">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="28.64%" id="mcps1.3.2.3.2.2.4.1.3"><p id="mrs_01_1981__ad978889e29584754bc6633608d229ce1"><strong id="mrs_01_1981__a2f0888506e384097b8fdab1adfff2eaf">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1981__ree361911664044218173ace3515f858e"><td class="cellrowborder" valign="top" width="29.759999999999998%" headers="mcps1.3.2.3.2.2.4.1.1 "><p id="mrs_01_1981__a846223535dec408997f033911a827414">spark.dynamicAllocation.minExecutors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.6%" headers="mcps1.3.2.3.2.2.4.1.2 "><p id="mrs_01_1981__a36597f30acb5494b8aea0ac1aa5ad50c">Indicates the minimum number of executors.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.64%" headers="mcps1.3.2.3.2.2.4.1.3 "><p id="mrs_01_1981__a3d3f765a6ac046c08f3cb85b399331dc">0</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1981__r47399f7d59b34f6ea1a926b92bb0d3f2"><td class="cellrowborder" valign="top" width="29.759999999999998%" headers="mcps1.3.2.3.2.2.4.1.1 "><p id="mrs_01_1981__a8a82259ac7204b4ca246ab304c69144a">spark.dynamicAllocation.initialExecutors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.6%" headers="mcps1.3.2.3.2.2.4.1.2 "><p id="mrs_01_1981__ab0009345633b48ec8a65ce439f28e407">Indicates the number of initial executors.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.64%" headers="mcps1.3.2.3.2.2.4.1.3 "><p id="mrs_01_1981__a7a7445c8ab4a4d458dec08c9140a84ca">0</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1981__r26086e2d92564e369bf27becc8e10382"><td class="cellrowborder" valign="top" width="29.759999999999998%" headers="mcps1.3.2.3.2.2.4.1.1 "><p id="mrs_01_1981__ac7448781814f408d938133d2b07d06cc">spark.dynamicAllocation.maxExecutors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.6%" headers="mcps1.3.2.3.2.2.4.1.2 "><p id="mrs_01_1981__ad42b416b21af4e5f93f38d4b49dbd64d">Indicates the maximum number of executors.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.64%" headers="mcps1.3.2.3.2.2.4.1.3 "><p id="mrs_01_1981__a1a255a6b66984fc09645b2fd9bd6c6f8">2048</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1981__rf0d2a6f810bc42718271fced780451ef"><td class="cellrowborder" valign="top" width="29.759999999999998%" headers="mcps1.3.2.3.2.2.4.1.1 "><p id="mrs_01_1981__aac1fd05e6f2f4d839ccdf6ae00193f7a">spark.dynamicAllocation.schedulerBacklogTimeout</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.6%" headers="mcps1.3.2.3.2.2.4.1.2 "><p id="mrs_01_1981__a00c3b3a003a2460d94d952d3a5da01fd">Indicates the first timeout period for scheduling.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.64%" headers="mcps1.3.2.3.2.2.4.1.3 "><p id="mrs_01_1981__a5e83db8c0040472bbe58d23a17ff7417">1s</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1981__rd6c8f8673f6449c0b1479bed919aa570"><td class="cellrowborder" valign="top" width="29.759999999999998%" headers="mcps1.3.2.3.2.2.4.1.1 "><p id="mrs_01_1981__aee6f79decf6545cea8a840fbbe9a7874">spark.dynamicAllocation.sustainedSchedulerBacklogTimeout</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.6%" headers="mcps1.3.2.3.2.2.4.1.2 "><p id="mrs_01_1981__a59d47183960146ad9468335d5075ce5c">Indicates the second and later timeout interval for scheduling.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.64%" headers="mcps1.3.2.3.2.2.4.1.3 "><p id="mrs_01_1981__a7c469dfcbbe44c5d81d61c4538d534f6">1s</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1981__r2c0ab5e58caa4181bef1811cb3d0299d"><td class="cellrowborder" valign="top" width="29.759999999999998%" headers="mcps1.3.2.3.2.2.4.1.1 "><p id="mrs_01_1981__a046a1c26d06841f9bc4ce987560af9ae">spark.dynamicAllocation.executorIdleTimeout</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.6%" headers="mcps1.3.2.3.2.2.4.1.2 "><p id="mrs_01_1981__aa5c3f2a91e134889868df5e0cce8977b">Indicates the idle timeout interval for common executors.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.64%" headers="mcps1.3.2.3.2.2.4.1.3 "><p id="mrs_01_1981__ab1ffd6bdca2f4326accb62537ffb8aee">60s</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1981__r0e120cf4216140b6a19a12885a45f5e7"><td class="cellrowborder" valign="top" width="29.759999999999998%" headers="mcps1.3.2.3.2.2.4.1.1 "><p id="mrs_01_1981__aeade2347c4f54f4daa0ca76f34416a2d">spark.dynamicAllocation.cachedExecutorIdleTimeout</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.6%" headers="mcps1.3.2.3.2.2.4.1.2 "><p id="mrs_01_1981__acade975c56c54e5ba3a20f907a6f76fe">Indicates the idle timeout interval for executors with cached blocks.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="28.64%" headers="mcps1.3.2.3.2.2.4.1.3 "><ul id="mrs_01_1981__ul756811528278"><li id="mrs_01_1981__li13568115282714">JDBCServer2x: 2147483647s</li><li id="mrs_01_1981__li0159203201118">IndexServer2x: 2147483647s</li><li id="mrs_01_1981__li1256917522270">SparkResource2x: 120</li></ul>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_1981__n0aaed639ca7d4970897fa5b67ac276f9"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1981__ae466d32fc4b94a3cb4812e32845f1860">The external shuffle service must be configured before using the dynamic resource scheduling function.</p>
|
|
</div></div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1975.html">Spark Core Tuning</a></div>
|
|
</div>
|
|
</div>
|
|
|