forked from docs/doc-exports
Yang, Tong
3b1f73dece
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
90 lines
14 KiB
HTML
90 lines
14 KiB
HTML
<a name="ALM-18008"></a><a name="ALM-18008"></a>
|
|
|
|
<h1 class="topictitle1">ALM-18008 Heap Memory Usage of ResourceManager Exceeds the Threshold</h1>
|
|
<div id="body43131373"><div class="section" id="ALM-18008__sc6dd350dd91b44258a56a434cfcc5291"><h4 class="sectiontitle">Description</h4><p id="ALM-18008__en-us_topic_0070543505_p8060118">The system checks the heap memory usage of Yarn ResourceManager every 30 seconds and compares the actual usage with the threshold. The alarm is generated when the heap memory usage of Yarn ResourceManager exceeds the threshold (95% of the maximum memory by default).</p>
|
|
<p id="ALM-18008__en-us_topic_0070543505_p5432205">Users can choose <strong id="ALM-18008__b10464182514124">O&M > Alarm > </strong><strong id="ALM-18008__b122075817202">Thresholds</strong> > <em id="ALM-18008__i1021725095817">Name of the desired cluster</em> > <strong id="ALM-18008__en-us_topic_0070543505_b5894231">Yarn</strong> to change the threshold.</p>
|
|
<p id="ALM-18008__p603214059351">When the <strong id="ALM-18008__b48421890111935">Trigger Count</strong> is 1, this alarm is cleared when the heap memory usage of Yarn ResourceManager is less than or equal to the threshold. When the <strong id="ALM-18008__b849810512394">Trigger Count</strong> is greater than 1, this alarm is cleared when the heap memory usage of Yarn ResourceManager is less than or equal to 95% of the threshold.</p>
|
|
</div>
|
|
<div class="section" id="ALM-18008__s647d6c22be4148a6bdceba47d66dc6c0"><h4 class="sectiontitle">Attribute</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-18008__en-us_topic_0070543505_table1927776" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-18008__en-us_topic_0070543505_row15537542"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.1"><p id="ALM-18008__en-us_topic_0070543505_p50581426">Alarm ID</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.2"><p id="ALM-18008__en-us_topic_0070543505_p3454809">Alarm Severity</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.3"><p id="ALM-18008__en-us_topic_0070543505_p11404133">Automatically Cleared</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="ALM-18008__en-us_topic_0070543505_row51319614"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.1 "><p id="ALM-18008__en-us_topic_0070543505_p63248075">18008</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.2 "><p id="ALM-18008__en-us_topic_0070543505_p22820469">Major</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.3 "><p id="ALM-18008__en-us_topic_0070543505_p36518691">Yes</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="ALM-18008__s28f94c493278466db335e78ffc82c729"><h4 class="sectiontitle">Parameters</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-18008__en-us_topic_0070543505_table5223998" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-18008__en-us_topic_0070543505_row39530145"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.1"><p id="ALM-18008__en-us_topic_0070543505_p47825138">Name</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.2"><p id="ALM-18008__en-us_topic_0070543505_p48630964">Meaning</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="ALM-18008__row550435111228"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18008__p192431315431">Source</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18008__p692551319435">Specifies the cluster for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="ALM-18008__en-us_topic_0070543505_row46793998"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18008__en-us_topic_0070543505_p32217513">ServiceName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18008__en-us_topic_0070543505_p59481773">Specifies the service name for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="ALM-18008__en-us_topic_0070543505_row65573909"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18008__en-us_topic_0070543505_p9886408">RoleName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18008__en-us_topic_0070543505_p62601592">Specifies the role name for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="ALM-18008__en-us_topic_0070543505_row26543418"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18008__en-us_topic_0070543505_p2533231">HostName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18008__en-us_topic_0070543505_p3865173">Specifies the object (host ID) for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="ALM-18008__en-us_topic_0070543505_row34786562"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18008__en-us_topic_0070543505_p66248115">Trigger Condition</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18008__en-us_topic_0070543505_p64497130">Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="ALM-18008__s11ebeeb9e1ed4bdcb68a31fedd407a97"><h4 class="sectiontitle">Impact on the System</h4><p id="ALM-18008__en-us_topic_0070543505_p56885004">When the heap memory usage of Yarn ResourceManager is overhigh, the performance of Yarn task submission and operation is affected. In addition, a memory overflow may occur so that the Yarn service is unavailable.</p>
|
|
</div>
|
|
<div class="section" id="ALM-18008__sf0dc654677fa4d089bcbe283f1683f8e"><h4 class="sectiontitle">Possible Causes</h4><p id="ALM-18008__en-us_topic_0070543505_p44282627">The heap memory of the Yarn ResourceManager instance on the node is overused or the heap memory is inappropriately allocated. As a result, the usage exceeds the threshold.</p>
|
|
</div>
|
|
<div class="section" id="ALM-18008__s40afa7da76044523adf322921e31d9c9"><h4 class="sectiontitle">Procedure</h4><p class="tableheading" id="ALM-18008__en-us_topic_0070543505_p30123063"><strong id="ALM-18008__b170160418521">Check the heap memory usage.</strong></p>
|
|
<ol id="ALM-18008__ol5562853218528"><li id="ALM-18008__li6697326185157"><span>On the FusionInsight Manager portal, choose <strong id="ALM-18008__b49995624185157">O&M > Alarm<strong id="ALM-18008__b27872374104950"> > Alarms</strong></strong> > <strong id="ALM-18008__b47307439185157">Heap Memory Usage of Yarn ResourceManager Exceeds the Threshold</strong> > <strong id="ALM-18008__b23113768185157">Location</strong>. Check the HostName of the instance for which the alarm is generated.</span></li><li id="ALM-18008__li30241261185157"><span>On the FusionInsight Manager portal, choose <strong id="ALM-18008__b14368410417">Cluster </strong>> <em id="ALM-18008__i243934194113">Name of the desired cluster</em><strong id="ALM-18008__b443774164117"> > Services</strong> > <strong id="ALM-18008__b5612522185157">Yarn</strong> > <strong id="ALM-18008__b50512698185157">Instance</strong> > <strong id="ALM-18008__b484421623815">ResourceManager</strong> (Indicates the host name of the instance for which the alarm is generated)<strong id="ALM-18008__b196611230382">.</strong> Click the drop-down menu in the upper right corner of <strong id="ALM-18008__b3273144141318">Chart</strong>, choose <strong id="ALM-18008__b7246166191312">Customize</strong> > <strong id="ALM-18008__b249047163812">ResourceManager > </strong><strong id="ALM-18008__b9145313427">Percentage of Used Memory of the ResourceManager</strong>. Check the heap memory usage.</span></li><li id="ALM-18008__li14085206185157"><span>Check whether the used heap memory of ResourceManager reaches 95% of the maximum heap memory specified for ResourceManager.</span><p><ul class="subitemlist" id="ALM-18008__ul16478103185157"><li id="ALM-18008__li33623063185157">If yes, go to <a href="#ALM-18008__li36040348185157">4</a>.</li><li id="ALM-18008__li39113602185157">If no, go to <a href="#ALM-18008__li63055176185157">6</a>.</li></ul>
|
|
</p></li><li id="ALM-18008__li36040348185157"><a name="ALM-18008__li36040348185157"></a><a name="li36040348185157"></a><span>On the FusionInsight Manager portal, choose <strong id="ALM-18008__b10411184413">Cluster </strong>> <em id="ALM-18008__i351018124111">Name of the desired cluster</em><strong id="ALM-18008__b104318104112"> > Services</strong> > <strong id="ALM-18008__b51001185157">Yarn</strong> > <strong id="ALM-18008__b459017185157">Configurations</strong> > <strong id="ALM-18008__b4131158185157">All</strong> <strong id="ALM-18008__b138561848454">Configurations </strong>> <strong id="ALM-18008__b37180425185157">ResourceManager</strong> > <strong id="ALM-18008__b66188371185157">System</strong>. Increase the value of <strong id="ALM-18008__b59657816185157">GC_OPTS</strong> parameter as required, click <strong id="ALM-18008__b49438185157">Save</strong>. Restart the role instance.</span><p><div class="note" id="ALM-18008__note287927164119"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="ALM-18008__p2879373416">The mapping between the number of NodeManager instances in a cluster and the memory size of ResourceManager is as follows:</p>
|
|
<ul id="ALM-18008__ul4792102219417"><li id="ALM-18008__li1679215224416">If the number of NodeManager instances in the cluster reaches 100, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=1G</li><li id="ALM-18008__li9792152219417">If the number of NodeManager instances in the cluster reaches 200, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms6G -Xmx6G -XX:NewSize=512M -XX:MaxNewSize=1G</li><li id="ALM-18008__li1792182214414">If the number of NodeManager instances in the cluster reaches 500, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms10G -Xmx10G -XX:NewSize=1G -XX:MaxNewSize=2G</li><li id="ALM-18008__li107931322134120">If the number of NodeManager instances in the cluster reaches 1000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms20G -Xmx20G -XX:NewSize=1G -XX:MaxNewSize=2G</li><li id="ALM-18008__li779312229417">If the number of NodeManager instances in the cluster reaches 2000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms40G -Xmx40G -XX:NewSize=2G -XX:MaxNewSize=4G</li><li id="ALM-18008__li7793102214119">If the number of NodeManager instances in the cluster reaches 3000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms60G -Xmx60G -XX:NewSize=2G -XX:MaxNewSize=4G</li><li id="ALM-18008__li979332218413">If the number of NodeManager instances in the cluster reaches 4000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms80G -Xmx80G -XX:NewSize=2G -XX:MaxNewSize=4G</li><li id="ALM-18008__li57936227415">If the number of NodeManager instances in the cluster reaches 5000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms100G -Xmx100G -XX:NewSize=3G -XX:MaxNewSize=6G</li></ul>
|
|
</div></div>
|
|
</p></li><li id="ALM-18008__li46666859185157"><span>Check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-18008__ul36201827185157"><li id="ALM-18008__li55927684185157">If yes, no further action is required.</li><li id="ALM-18008__li33848587185157">If no, go to <a href="#ALM-18008__li63055176185157">6</a>.</li></ul>
|
|
</p></li></ol>
|
|
<p class="tableheading" id="ALM-18008__p57380989185157"><strong id="ALM-18008__b58111632185217">Collect fault information.</strong></p>
|
|
<ol start="6" id="ALM-18008__ol42167917185220"><li id="ALM-18008__li63055176185157"><a name="ALM-18008__li63055176185157"></a><a name="li63055176185157"></a><span>On the FusionInsight Manager portal, choose <strong id="ALM-18008__b9513753886">O&M</strong> > <strong id="ALM-18008__b21919211185157">Log > Download</strong>.</span></li><li id="ALM-18008__li33014478185157"><span>Select the following node in the required cluster from the <strong id="ALM-18008__b30625676185157">Service</strong>.</span><p><ul class="subitemlist" id="ALM-18008__ul11124815185157"><li id="ALM-18008__li64760693185157">NodeAgent</li><li id="ALM-18008__li45975333185157">Yarn</li></ul>
|
|
</p></li><li id="ALM-18008__li1145664103113"><span>Click <span><img id="ALM-18008__image1945644173117" src="en-us_image_0269417395.png"></span> in the upper right corner, and set <strong id="ALM-18008__b6456941173117">Start Date</strong> and <strong id="ALM-18008__b11456154113318">End Date</strong> for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click <strong id="ALM-18008__b13456164113319">Download</strong>.</span></li><li id="ALM-18008__li37613332185157"><span>Contact the <span id="ALM-18008__text4614151421417">O&M personnel</span> and send the collected logs.</span></li></ol>
|
|
</div>
|
|
<div class="section" id="ALM-18008__section1529716184534"><h4 class="sectiontitle">Alarm Clearing</h4><p id="ALM-18008__p4677152685316">After the fault is rectified, the system automatically clears this alarm.</p>
|
|
</div>
|
|
<div class="section" id="ALM-18008__s1a69b00ec8d042d797145d10db6468db"><h4 class="sectiontitle">Related Information</h4><p id="ALM-18008__en-us_topic_0070543505_p48266778">None</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1298.html">Alarm Reference (Applicable to MRS 3.x)</a></div>
|
|
</div>
|
|
</div>
|
|
|