doc-exports/docs/mrs/component-operation-guide/mrs_01_0872.html

<a name="mrs_01_0872"></a><a name="mrs_01_0872"></a>

<h1 class="topictitle1">Preempting a Task</h1>
<div id="body1590130748022"><div class="section" id="mrs_01_0872__s6a98c6e0860e4a4f8578aa5cd80fdb53"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0872__a18c48eb1a67a46c59be4ac6b0a89ecf6">The capacity scheduler of ResourceManager implements job preemption to simplify job running in queues and improve resource utilization. The process is as follows:</p>
<ol id="mrs_01_0872__oaf54a4cb91f94a69a1f552ae7891cced"><li id="mrs_01_0872__l71631502dd544bf38c33a50472f03d1f">Assume that there are two queues (Queue A and Queue B). The capacity of Queue A is 25%, and the capacity of Queue B is 75%.</li><li id="mrs_01_0872__lc3229a9025444449a5e9558aaa3a14c9">In the initial state, Task 1 is distributed to Queue A for processing, requiring 75% cluster resources. Task 2 is distributed to Queue B for processing, requiring 50% cluster resources.</li><li id="mrs_01_0872__l387c86a4b8ff43a1929b79fea6c0c826">Task 1 uses 25% cluster resources provided by Queue A and 50% resources from Queue B. Queue B reserves 25% cluster resources.</li><li id="mrs_01_0872__l72dd1fe44d8640b9b34dd24539e7da44">If task preemption is enabled, the resources of Task 1 will be preempted. Queue B preempts 25% cluster resources from Queue A for Task 2.</li><li id="mrs_01_0872__l3a6ab871046045669cb6430e0b8b7fc0">Task 1 will be executed when Task 2 is complete and the cluster has sufficient resources.</li></ol>
</div>
<div class="section" id="mrs_01_0872__s7b8212acfdec471593ef1109d51a35ab"><h4 class="sectiontitle">Procedure</h4><p id="mrs_01_0872__ae4d320c1554544c48876febf2d94dc18">Navigation path for setting parameters:</p>
<p id="mrs_01_0872__p165569341183">Go to the <strong id="mrs_01_0872__b710122283820">All Configurations</strong> page of Yarn and enter a parameter name in the search box by referring to <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.</p>

<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0872__tc91b23bb26c248cba7fde1cd48dc09af" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0872__r2b0987648bab40559127c07bde30a0d6"><th align="left" class="cellrowborder" valign="top" width="31.830000000000002%" id="mcps1.3.2.4.2.4.1.1"><p id="mrs_01_0872__a169915bd732d4d6a9b9beb68e0ca27dd">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50.81%" id="mcps1.3.2.4.2.4.1.2"><p id="mrs_01_0872__acff9a87a645748b5abc52606ac07dbb7">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="17.36%" id="mcps1.3.2.4.2.4.1.3"><p id="mrs_01_0872__a077736c116944ed4bb0c92f694cc8480">Default Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_0872__rc6ce9a4ab35c4850a9fa24d6ed008416"><td class="cellrowborder" valign="top" width="31.830000000000002%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0872__a6f6d19de7cd3492b9fb2943048371fc5">yarn.resourcemanager.scheduler.monitor.enable</p>
</td>
<td class="cellrowborder" valign="top" width="50.81%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0872__a28df24d9dff54b6296003c641b38e7b7">Whether to start scheduler monitoring according to <strong id="mrs_01_0872__b13680161564418">yarn.resourcemanager.scheduler.monitor.policies</strong>. If this parameter is set to <strong id="mrs_01_0872__b17228123116449">true</strong>, scheduler monitoring is enabled based on policies specified by <strong id="mrs_01_0872__b1694915445441">yarn.resourcemanager.scheduler.monitor.policies</strong> and task resource preemption is enabled based on the scheduler information. If this parameter is set to <strong id="mrs_01_0872__b14619818454">false</strong>, scheduler monitoring is disabled.</p>
</td>
<td class="cellrowborder" valign="top" width="17.36%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0872__a583a1f3e57764bb6bc93e69236351ad8">false</p>
</td>
</tr>
<tr id="mrs_01_0872__ra743b8b52a7044bfbd6d606409ef1f24"><td class="cellrowborder" valign="top" width="31.830000000000002%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0872__af04112c6c87a4fa2becc876ac6612447">yarn.resourcemanager.scheduler.monitor.policies</p>
</td>
<td class="cellrowborder" valign="top" width="50.81%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0872__a1ee9f9b4def649f68f50b14101e69533">List of the SchedulingEditPolicy class to be used with the scheduler</p>
</td>
<td class="cellrowborder" valign="top" width="17.36%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0872__ac7f43adbfc0b4bbeb4c099e77b6cac39">org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</p>
</td>
</tr>
<tr id="mrs_01_0872__r06f98657d7e14d108f57db2a52814a45"><td class="cellrowborder" valign="top" width="31.830000000000002%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0872__a05b64da0f52d49938fab973efa446219">yarn.resourcemanager.monitor.capacity.preemption.observe_only</p>
</td>
<td class="cellrowborder" valign="top" width="50.81%" headers="mcps1.3.2.4.2.4.1.2 "><ul id="mrs_01_0872__u1d98c9a315de46dfb217130b4babe697"><li id="mrs_01_0872__lebef8eab0fd144148797b2e20963a6ef">If this parameter is set to <strong id="mrs_01_0872__b05521546124520">true</strong>, policies will be applied but task resource preemption will not be performed.</li><li id="mrs_01_0872__l8a4bd029c2f346738a6647ad2ff664f6">If this parameter is set to <strong id="mrs_01_0872__b9334135194617">false</strong>, policies will be applied and task resource preemption will be performed based on the policies.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="17.36%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0872__afbc5af01c187408dadc14c639cd26f85">false</p>
</td>
</tr>
<tr id="mrs_01_0872__rd5a74bc2b06240dabeceaaab2a3d108d"><td class="cellrowborder" valign="top" width="31.830000000000002%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0872__af0f85898721943e3b4cee693e825c21d">yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval</p>
</td>
<td class="cellrowborder" valign="top" width="50.81%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0872__a66b9ef69e47541fbb994b75a1a6fce77">Monitoring interval, in millisecond. If this parameter is set to a larger value, capacity detection will not be performed frequently.</p>
</td>
<td class="cellrowborder" valign="top" width="17.36%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0872__a0bcd752acf2841848047b620eed5def2">3000</p>
</td>
</tr>
<tr id="mrs_01_0872__r3e397035497545b0b4ca040d1ca1b3e8"><td class="cellrowborder" valign="top" width="31.830000000000002%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0872__a0698ba9790a547ec8a79f1bac53e89d1">yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill</p>
</td>
<td class="cellrowborder" valign="top" width="50.81%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0872__a3bd8d3ebfedd4bb297d57a75dd293374">Interval between the time when a resource preemption request is sent and the time when the container is stopped (resources are released), in millisecond. The value must be greater than or equal to <strong id="mrs_01_0872__b15901541204711">0</strong>.</p>
<p id="mrs_01_0872__afaf171599a444767a7cbd762e45c4a73">By default, if ApplicationMaster does not stop the container within 15 seconds, ResourceManager will forcibly stop the container after 15 seconds.</p>
</td>
<td class="cellrowborder" valign="top" width="17.36%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0872__a173e3173dce94aedbf04a36a4b77129c">15000</p>
</td>
</tr>
<tr id="mrs_01_0872__r7dcc3723a14e4ba68a150aa28f96f39a"><td class="cellrowborder" valign="top" width="31.830000000000002%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0872__ab8b3aa4e2efe403d9acc7c741c8e705d">yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round</p>
</td>
<td class="cellrowborder" valign="top" width="50.81%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0872__a56f274b38c0c4b4b9cc53e722f3a0580">Maximum resource preemption ratio in a period. This value can be used to limit the speed at which containers are reclaimed from the cluster. After the expected total preemption value is calculated, the policy scales the preemption ratio back to this limit.</p>
</td>
<td class="cellrowborder" valign="top" width="17.36%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0872__a1f34b0eaedb3466fac235d1c3c80b24b">0.1</p>
</td>
</tr>
<tr id="mrs_01_0872__r2f0f11c68dbd43b387d07dd61a6d022d"><td class="cellrowborder" valign="top" width="31.830000000000002%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0872__ad354e8e3d23b4b06a02c5d639b5bb9b7">yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity</p>
</td>
<td class="cellrowborder" valign="top" width="50.81%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0872__ac27a564fa7b34e7e9236097c1a78848e">Resource preemption dead zone = Total number of resources in the cluster x Value of this configuration item + Original resources of a queue (for example, Queue A). When resources actually used by a task in Queue A exceeds the preemption dead zone, the resource beyond the preemption dead zone is preempted. The value range is 0 to 1.</p>
<div class="note" id="mrs_01_0872__ne0f35348d7ed4f00be299469e5e52fac"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_0872__a8dbe5419fc974a2eb459ffa8f30bde14">A smaller value is recommended for effective preemption.</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="17.36%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0872__abe462e20df5f4940976b86f7a06b6396">0</p>
</td>
</tr>
<tr id="mrs_01_0872__r30ceb78c7d2746a785cd900fef3723d4"><td class="cellrowborder" valign="top" width="31.830000000000002%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0872__ad711be5576fb4fb2a0562ce232039c20">yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor</p>
</td>
<td class="cellrowborder" valign="top" width="50.81%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0872__a7896ffa3a7cb45d486961c5c5ce9a2b9">Preemption percentage. Containers preempt only this percentage of the resources.</p>
<p id="mrs_01_0872__af5313843aca04f02bf51bb29d1605fc5">For example, a termination factor of 0.5 will reclaim almost 95% of resources within 5 times of <strong id="mrs_01_0872__b77713558541">yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill</strong>, even in the absence of natural termination. That is, 5 consecutive preemptions will be performed and each time half of the target resources will be preempted. The trend is geometric convergence. The interval of each preemption is <span class="parmname" id="mrs_01_0872__pea57da3078d7431c9e5c6558b110a1b6"><b>yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill</b></span>. The value range is 0 to 1.</p>
</td>
<td class="cellrowborder" valign="top" width="17.36%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0872__a8c1d9be992494ba6bdc726875c1a7118">1</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0871.html">Yarn Performance Tuning</a></div>
</div>
</div>