forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
77 lines
9.6 KiB
HTML
77 lines
9.6 KiB
HTML
<a name="mrs_01_0861"></a><a name="mrs_01_0861"></a>
|
||
|
||
<h1 class="topictitle1">Configure the ApplicationMaster to Automatically Adjust the Allocated Memory</h1>
|
||
<div id="body1590130746699"><p id="mrs_01_0861__p4431810205817">This section applies to clusters of MRS 3.<em id="mrs_01_0861__i38681217183610">x</em> or later.</p>
|
||
<div class="section" id="mrs_01_0861__s6769d5d3f49b4b13a1d1021094ec5270"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0861__ae580b5afe29543ff92531ee434e166b2">During the process of starting the configuration, when the ApplicationMaster creates a container, the allocated memory is automatically adjusted according to the total number of tasks, which makes resource utilization more flexible and improves the fault tolerance of the client application.</p>
|
||
</div>
|
||
<div class="section" id="mrs_01_0861__section3730112261910"><h4 class="sectiontitle">Configuration Description</h4><p id="mrs_01_0861__a92550ce6bd694b7db0faeb54173387c0"><strong id="mrs_01_0861__a9cc8efdc2b5a43d78458a4c79484c11a">Navigation path for setting parameters:</strong></p>
|
||
<p id="mrs_01_0861__af28a32f7c205411f8fd64cbcb75f91aa">On Manager, choose <strong id="mrs_01_0861__b83656286642326">Cluster</strong> > <em id="mrs_01_0861__i101411628442326">Name of the desired cluster</em> > <strong id="mrs_01_0861__b5202371742326">Service</strong> > <strong id="mrs_01_0861__b195665959042326">Yarn</strong> > <strong id="mrs_01_0861__b58454883142326">Configuration</strong>. On the displayed page, select <span class="parmvalue" id="mrs_01_0861__parmvalue11901674491"><b>All Configurations</b></span> and enter <strong id="mrs_01_0861__b111857902842326">mapreduce.job.am.memory.policy</strong>.</p>
|
||
<p id="mrs_01_0861__p11461151712221"><strong id="mrs_01_0861__b98603172519">Configuration description</strong></p>
|
||
<p id="mrs_01_0861__p1964910297309">If the default value of the parameter is left empty. In this case, the automatic adjustment policy is not enabled. The memory of ApplicationMaster is still affected by the value of <strong id="mrs_01_0861__b15004432942326">yarn.app.mapreduce.am.resource.mb</strong>.</p>
|
||
<p id="mrs_01_0861__p201502027182510">The value of <strong id="mrs_01_0861__b140948245642326">mapreduce.job.am.memory.policy</strong> consists of five items, and they are separated by colons (:) and commas (,) in the following format: <strong id="mrs_01_0861__b253985013281">baseTaskCount:taskStep:memoryStep,minMemory:maxMemory</strong>. The format is strictly checked when the value is entered.</p>
|
||
|
||
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0861__table7697021382" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0861__row16978293819"><th align="left" class="cellrowborder" valign="top" width="10.611061106110611%" id="mcps1.3.3.7.2.4.1.1"><p id="mrs_01_0861__p196971226383">Parameter</p>
|
||
</th>
|
||
<th align="left" class="cellrowborder" valign="top" width="59.94599459945995%" id="mcps1.3.3.7.2.4.1.2"><p id="mrs_01_0861__p96971724383">Description</p>
|
||
</th>
|
||
<th align="left" class="cellrowborder" valign="top" width="29.44294429442944%" id="mcps1.3.3.7.2.4.1.3"><p id="mrs_01_0861__p869719214386">Setting Requirement</p>
|
||
</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody><tr id="mrs_01_0861__row969716293819"><td class="cellrowborder" valign="top" width="10.611061106110611%" headers="mcps1.3.3.7.2.4.1.1 "><p id="mrs_01_0861__p11697525381">baseTaskCount</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="59.94599459945995%" headers="mcps1.3.3.7.2.4.1.2 "><p id="mrs_01_0861__p669722103814">Indicates the total number of tasks. The configuration of ApplicationMaster is valid only when the total number of tasks (on the sum of the Map and Reduce ends) is greater than or equal to the value of this parameter.</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="29.44294429442944%" headers="mcps1.3.3.7.2.4.1.3 "><p id="mrs_01_0861__p86971253815">The value cannot be empty and must be greater than 0.</p>
|
||
</td>
|
||
</tr>
|
||
<tr id="mrs_01_0861__row18697227387"><td class="cellrowborder" valign="top" width="10.611061106110611%" headers="mcps1.3.3.7.2.4.1.1 "><p id="mrs_01_0861__p069812233812">taskStep</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="59.94599459945995%" headers="mcps1.3.3.7.2.4.1.2 "><p id="mrs_01_0861__p136981321383">Indicates the incremental step length of tasks. This parameter and <strong id="mrs_01_0861__b131526975542326">memoryStep</strong> determine the memory adjustment amount.</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="29.44294429442944%" headers="mcps1.3.3.7.2.4.1.3 "><p id="mrs_01_0861__p269852133819">The value cannot be empty and must be greater than 0.</p>
|
||
</td>
|
||
</tr>
|
||
<tr id="mrs_01_0861__row16985223816"><td class="cellrowborder" valign="top" width="10.611061106110611%" headers="mcps1.3.3.7.2.4.1.1 "><p id="mrs_01_0861__p166984211383">memoryStep</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="59.94599459945995%" headers="mcps1.3.3.7.2.4.1.2 "><p id="mrs_01_0861__p169872183819">Indicates the incremental memory step. The memory capacity is increased based on the value of <strong id="mrs_01_0861__b21821644942326">yarn.app.mapreduce.am.resource.mb</strong>.</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="29.44294429442944%" headers="mcps1.3.3.7.2.4.1.3 "><p id="mrs_01_0861__p14698112123818">The value cannot be empty and must be greater than 0. The unit is MB.</p>
|
||
</td>
|
||
</tr>
|
||
<tr id="mrs_01_0861__row1369817214383"><td class="cellrowborder" valign="top" width="10.611061106110611%" headers="mcps1.3.3.7.2.4.1.1 "><p id="mrs_01_0861__p116985253815">minMemory</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="59.94599459945995%" headers="mcps1.3.3.7.2.4.1.2 "><p id="mrs_01_0861__p16988218388">Indicates the lower limit of the memory that can be automatically adjusted. If the memory after the automatic adjustment is less than or equal to the value of this parameter, the value of <strong id="mrs_01_0861__b199197096142326">yarn.app.mapreduce.am.resource.mb</strong> is used.</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="29.44294429442944%" headers="mcps1.3.3.7.2.4.1.3 "><p id="mrs_01_0861__p11698828389">The value cannot be empty. It must be greater than 0 and cannot be greater than the value of <strong id="mrs_01_0861__b46893346942326">maxMemory</strong>.</p>
|
||
<p id="mrs_01_0861__p23901442192118">Unit: MB</p>
|
||
</td>
|
||
</tr>
|
||
<tr id="mrs_01_0861__row869816218382"><td class="cellrowborder" valign="top" width="10.611061106110611%" headers="mcps1.3.3.7.2.4.1.1 "><p id="mrs_01_0861__p10698928383">maxMemory</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="59.94599459945995%" headers="mcps1.3.3.7.2.4.1.2 "><p id="mrs_01_0861__p9698172103810">Indicates the upper limit of memory that can ve automatically adjusted. If the adjusted memory exceeds the upper limit, use this value as the final value.</p>
|
||
</td>
|
||
<td class="cellrowborder" valign="top" width="29.44294429442944%" headers="mcps1.3.3.7.2.4.1.3 "><p id="mrs_01_0861__p469842183811">The value cannot be empty. It must be greater than 0 and cannot be less than the value of <strong id="mrs_01_0861__b181786006942326">minMemory</strong>.</p>
|
||
<p id="mrs_01_0861__p329164812111">Unit: MB</p>
|
||
</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
</div>
|
||
<div class="section" id="mrs_01_0861__section18404650181920"><h4 class="sectiontitle">Example Value</h4><p id="mrs_01_0861__p940445091913">Configuration:</p>
|
||
<ul id="mrs_01_0861__ul78477818417"><li id="mrs_01_0861__li208479818417">yarn.app.mapreduce.am.resource.mb=1536</li><li id="mrs_01_0861__li49311512124114">mapreduce.job.am.memory.policy=100:10:50,1200:2000</li><li id="mrs_01_0861__li5112191817412">Total number of tasks of an application =120</li></ul>
|
||
<p id="mrs_01_0861__p13553101313251">The calculation process is as follows:</p>
|
||
<p id="mrs_01_0861__p174261648192519">Memory after adjustment = 1536 + [(120 – 100)/10] x 50 = 1636. In this example, memory after adjustment 1636 is greater than the value of <strong id="mrs_01_0861__b23209762042326">minMemory</strong> <strong id="mrs_01_0861__b26415057342326">1200</strong>, and less than the value of <strong id="mrs_01_0861__b129594626242326">maxMemory</strong> <strong id="mrs_01_0861__b156640482842326">2000</strong>. Therefore, the ApplicationMaster memory is set to <strong id="mrs_01_0861__b208166579642326">1636 MB</strong>.</p>
|
||
<p id="mrs_01_0861__p72143353310">If the value of <strong id="mrs_01_0861__b41382538142326">memStep</strong> is changed to <strong id="mrs_01_0861__b147123247342326">250</strong>, the calculation formula is as follows: Memory after adjustment = 1536 + [(120 – 100) / 10] x 250 = 2136. In this case, the memory after adjustment is greater than the value of <strong id="mrs_01_0861__b160027681942326">maxMemory</strong> <strong id="mrs_01_0861__b117719370442326">2000</strong>. As a result, the value of <strong id="mrs_01_0861__b22653852142326">ApplicationMaster</strong> is set to <strong id="mrs_01_0861__b140307136142326">2000 MB</strong>.</p>
|
||
<div class="note" id="mrs_01_0861__note9961319171"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0861__p696110192719">If the memory after adjustment is lower than the value of <strong id="mrs_01_0861__b199757475842326">minMemory</strong>, the configuration does not take effect but the value is still printed on the backend server. This value is provided as the reference for adjusting the value of <strong id="mrs_01_0861__b123595957542326">minMemory</strong>.</p>
|
||
</div></div>
|
||
</div>
|
||
</div>
|
||
<div>
|
||
<div class="familylinks">
|
||
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0851.html">Using Yarn</a></div>
|
||
</div>
|
||
</div>
|
||
|