vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/umn/ALM-18003.html
Yang, Tong 3b1f73dece MRS UMN 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-13 12:03:34 +00:00

94 lines
15 KiB
HTML
Raw Permalink Blame History

<a name="ALM-18003"></a><a name="ALM-18003"></a>
<h1 class="topictitle1">ALM-18003 NodeManager Unhealthy</h1>
<div id="body57672144"><div class="section" id="ALM-18003__s901b8a09c21d416aa2cad4ddac000513"><h4 class="sectiontitle">Description</h4><p id="ALM-18003__en-us_topic_0070543683_p66341581">The system checks the number of unhealthy NodeManager nodes every 30 seconds, and compares the number with the threshold. The Unhealthy Nodes indicator has a default threshold. This alarm is generated when the value of the Unhealthy Nodes indicator exceeds the threshold.</p>
<p id="ALM-18003__p16635714405">To change the threshold, on FusionInsight Manager, choose <strong id="ALM-18003__b81915107402">Cluster </strong>&gt; <em id="ALM-18003__i2020131094020">Name of the desired cluster</em> &gt; <strong id="ALM-18003__b963591415013">Services</strong> &gt; <strong id="ALM-18003__b063541418018">Yarn</strong>. On the displayed page, choose <strong id="ALM-18003__b116351914905">Configurations</strong> &gt; <strong id="ALM-18003__b56351414801">All Configurations</strong>, and change the value of <strong id="ALM-18003__b26351141303">yarn.nodemanager.unhealthy.alarm.threshold</strong>. You do not need to restart Yarn to make the change take effect.</p>
<p id="ALM-18003__p20635201410015">The default threshold is 0. The alarm is generated when the number of unhealthy nodes exceeds the threshold, and is cleared when the number of unhealthy nodes is less than the threshold.</p>
</div>
<div class="section" id="ALM-18003__s9eeafb3cd4e34921bb186751eb1d36ad"><h4 class="sectiontitle">Attribute</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-18003__en-us_topic_0070543683_table5404773" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-18003__en-us_topic_0070543683_row58805628"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.1"><p id="ALM-18003__en-us_topic_0070543683_p65635459">Alarm ID</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.2"><p id="ALM-18003__en-us_topic_0070543683_p14871925">Alarm Severity</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.3"><p id="ALM-18003__en-us_topic_0070543683_p63775292">Automatically Cleared</p>
</th>
</tr>
</thead>
<tbody><tr id="ALM-18003__en-us_topic_0070543683_row65525056"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.1 "><p id="ALM-18003__en-us_topic_0070543683_p5929320">18003</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.2 "><p id="ALM-18003__en-us_topic_0070543683_p10512925">Major</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.3 "><p id="ALM-18003__en-us_topic_0070543683_p46240582">Yes</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="ALM-18003__s47d1832e32a44bcc8be558c04b8adfe7"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-18003__en-us_topic_0070543683_table54499660" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-18003__en-us_topic_0070543683_row11966580"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.1"><p id="ALM-18003__en-us_topic_0070543683_p29768920">Name</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.2"><p id="ALM-18003__en-us_topic_0070543683_p62472349">Meaning</p>
</th>
</tr>
</thead>
<tbody><tr id="ALM-18003__row196761359132212"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18003__p192431315431">Source</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18003__p692551319435">Specifies the cluster for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-18003__en-us_topic_0070543683_row27095531"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18003__en-us_topic_0070543683_p47254377">ServiceName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18003__en-us_topic_0070543683_p2399295">Specifies the service for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-18003__en-us_topic_0070543683_row21593659"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18003__en-us_topic_0070543683_p4255945">RoleName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18003__en-us_topic_0070543683_p9187268">Specifies the role for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-18003__en-us_topic_0070543683_row15576549"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18003__en-us_topic_0070543683_p53740974">HostName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18003__en-us_topic_0070543683_p58051607">Specifies the host for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-18003__en-us_topic_0070543683_row52702415"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-18003__en-us_topic_0070543683_p41037195">Unhealthy Host</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-18003__p557818538018">Specifies the list of hosts with unhealthy nodes.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="ALM-18003__s301355fcb99e4f238cccd5ab0b9cf0fd"><h4 class="sectiontitle">Impact on the System</h4><ul id="ALM-18003__en-us_topic_0070543683_ul4279370"><li id="ALM-18003__en-us_topic_0070543683_li38514335">The faulty NodeManager node cannot provide the Yarn service.</li><li id="ALM-18003__en-us_topic_0070543683_li11084700">The number of containers decreases, so the cluster performance deteriorates.</li></ul>
</div>
<div class="section" id="ALM-18003__s69ec01e9dc174000a06b8a13414d9fa6"><h4 class="sectiontitle">Possible Causes</h4><ul id="ALM-18003__en-us_topic_0070543683_ul25445531"><li id="ALM-18003__en-us_topic_0070543683_li27683190">The hard disk space of the host where the NodeManager node resides is insufficient.</li><li id="ALM-18003__en-us_topic_0070543683_li47822126">User <strong id="ALM-18003__en-us_topic_0070543683_b27745955">omm</strong> does not have the permission to access a local directory on the NodeManager node.</li></ul>
</div>
<div class="section" id="ALM-18003__sc70c98d7f3cf4ad8900082b7c0408369"><h4 class="sectiontitle">Procedure</h4><p class="tableheading" id="ALM-18003__en-us_topic_0070543683_p32829872"><strong id="ALM-18003__b6690609518451">Check the hard disk space of the host.</strong></p>
<ol id="ALM-18003__ol66979456184511"><li id="ALM-18003__li23081553219"><span>On the FusionInsight Manager, and choose <strong id="ALM-18003__b6306135315120">O&amp;M</strong> &gt; <strong id="ALM-18003__b1930611537119">Alarm &gt; Alarms</strong>. Click <span><img id="ALM-18003__image1530618531417" src="en-us_image_0269417393.png"></span> before the alarm and obtain unhealthy nodes in <strong id="ALM-18003__b23061053310">Additional Information</strong>.</span></li><li id="ALM-18003__li11711432184458"><a name="ALM-18003__li11711432184458"></a><a name="li11711432184458"></a><span>Choose <strong id="ALM-18003__b586683313409">Cluster </strong>&gt; <em id="ALM-18003__i4870203354010">Name of the desired cluster</em><strong id="ALM-18003__b1886711330403"> &gt; Services &gt; Yarn</strong> &gt; <strong id="ALM-18003__b29630959184458">Instance</strong>, select the NodeManager instance corresponding to the host, choose<strong id="ALM-18003__b65352041184458"> Instance Configurations</strong><strong id="ALM-18003__b59023996184458"> &gt; All Configurations</strong> and view disks corresponding to <strong id="ALM-18003__b61453918184458">yarn.nodemanager.local-dirs</strong> and <strong id="ALM-18003__b16214351184458">yarn.nodemanager.log-dirs</strong>.</span></li><li id="ALM-18003__li60203709184458"><span>Choose <strong id="ALM-18003__b989423912514">O&amp;M &gt; Alarm</strong><strong id="ALM-18003__b27872374104950"> &gt; Alarms</strong>. In the alarm list, check whether the related disk has the alarm <strong id="ALM-18003__b9101918184458">ALM-12017 Insufficient Disk Capacity</strong>.</span><p><ul class="subitemlist" id="ALM-18003__ul51428543184458"><li id="ALM-18003__li66166794184458">If yes, go to <a href="#ALM-18003__li32014413184458">4</a>.</li><li id="ALM-18003__li57910065184458">If no, go to <a href="#ALM-18003__li64222725184458">5</a>.</li></ul>
</p></li><li id="ALM-18003__li32014413184458"><a name="ALM-18003__li32014413184458"></a><a name="li32014413184458"></a><span>Rectify the disk fault based on <strong id="ALM-18003__b5645102111717">ALM-12017 Insufficient Disk Capacit</strong> and check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-18003__ul11013697184458"><li id="ALM-18003__li44662294184458">If yes, no further action is required.</li><li id="ALM-18003__li60876067184458">If no, go to <a href="#ALM-18003__li4571214184458">7</a>.</li></ul>
</p></li><li id="ALM-18003__li64222725184458"><a name="ALM-18003__li64222725184458"></a><a name="li64222725184458"></a><span>Choose <strong id="ALM-18003__b20749143820216">Hosts</strong> &gt; <em id="ALM-18003__i145516203229">Name of the desired host . </em>On the <strong id="ALM-18003__b78881635112518">Dashboard</strong> page, check the disk usage of the corresponding partition. Check whether the percentage of the used space of the mounted disk exceeds the value of <strong id="ALM-18003__b1741817111731">yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</strong></span><p><ul class="subitemlist" id="ALM-18003__ul29505479184458"><li id="ALM-18003__li43030651184458">If yes, go to <a href="#ALM-18003__li54043174184458">6</a>.</li><li id="ALM-18003__li62930710184458">If no, go to <a href="#ALM-18003__li4571214184458">7</a>.</li></ul>
</p></li><li id="ALM-18003__li54043174184458"><a name="ALM-18003__li54043174184458"></a><a name="li54043174184458"></a><span>Reduce the disk usage to less than the value of <strong id="ALM-18003__b1770710533289">yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</strong>, wait for 10 to 20 minutes, and check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-18003__ul48720460184458"><li id="ALM-18003__li55852801184458">If yes, no further action is required.</li><li id="ALM-18003__li27783005184458">If no, go to <a href="#ALM-18003__li4571214184458">7</a>.</li></ul>
</p></li></ol>
<p class="tableheading" id="ALM-18003__p35830958184458"><strong id="ALM-18003__b5323160184521">Check the access permission of the local directory on each NodeManager node.</strong></p>
<ol start="7" id="ALM-18003__ol26775221184529"><li id="ALM-18003__li4571214184458"><a name="ALM-18003__li4571214184458"></a><a name="li4571214184458"></a><span>Obtain the NodeManager directory viewed in <a href="#ALM-18003__li11711432184458">2</a>, log in to each NodeManager node as user <strong id="ALM-18003__b15420993184458">root</strong>, and go to the obtained directory. <span id="ALM-18003__text427217203394"></span></span></li><li id="ALM-18003__li23458797184458"><span>Run the <strong id="ALM-18003__b41140933184458">ll</strong> command to check whether the permission of the <strong id="ALM-18003__b34724077184458">localdir</strong> and <strong id="ALM-18003__b44081240184458">containerlogs</strong> folders is <strong id="ALM-18003__b61186848184458">755</strong> and whether <strong id="ALM-18003__b13810725184458">User:Group</strong> is <strong id="ALM-18003__b57187664184458">omm:ficommon</strong>.</span><p><ul class="subitemlist" id="ALM-18003__ul2606533184458"><li id="ALM-18003__li44926930184458">If yes, no further action is required.</li><li id="ALM-18003__li15202695184458">If no, go to <a href="#ALM-18003__li55292474184458">9</a>.</li></ul>
</p></li><li id="ALM-18003__li55292474184458"><a name="ALM-18003__li55292474184458"></a><a name="li55292474184458"></a><span>Run the following command to set the permission to <strong id="ALM-18003__b9802586184458">755</strong> and <strong id="ALM-18003__b21114411184458">User:Group</strong> to <strong id="ALM-18003__b55811979184458">omm:ficommon</strong>:</span><p><p class="litext" id="ALM-18003__p18961746184458"><strong id="ALM-18003__b32545770184458">chmod 755 </strong><em id="ALM-18003__i24476482184458">&lt;folder_name&gt;</em></p>
<p class="litext" id="ALM-18003__p65795931184458"><strong id="ALM-18003__b36437988184458">chown omm:ficommon </strong><em id="ALM-18003__i59506442184458">&lt;folder_name&gt;</em></p>
</p></li><li id="ALM-18003__li65264558184458"><span>Wait for 10 to 20 minutes and check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-18003__ul50516004184458"><li id="ALM-18003__li27870223184458">If yes, no further action is required.</li><li id="ALM-18003__li42895591184458">If no, go to <a href="#ALM-18003__li64787675184458">11</a>.</li></ul>
</p></li></ol>
<p class="tableheading" id="ALM-18003__p32946582184534"><strong id="ALM-18003__b9717871184539">Collect fault information.</strong></p>
<ol start="11" id="ALM-18003__ol30743067184542"><li id="ALM-18003__li64787675184458"><a name="ALM-18003__li64787675184458"></a><a name="li64787675184458"></a><span>On the FusionInsight Manager in the active cluster, choose <strong id="ALM-18003__b11426165415523">O&amp;M</strong> &gt; <strong id="ALM-18003__b51937873184458">Log &gt; Download</strong>.</span></li><li id="ALM-18003__li4391931184458"><span>Select <strong id="ALM-18003__b46218164184458">Yarn</strong> in the required cluster from the <strong id="ALM-18003__b13310293184458">Service</strong>.</span></li><li id="ALM-18003__li1145664103113"><span>Click <span><img id="ALM-18003__image1945644173117" src="en-us_image_0269417394.png"></span> in the upper right corner, and set <strong id="ALM-18003__b6456941173117">Start Date</strong> and <strong id="ALM-18003__b11456154113318">End Date</strong> for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click <strong id="ALM-18003__b13456164113319">Download</strong>.</span></li><li id="ALM-18003__li6411914184458"><span>Contact the <span id="ALM-18003__text4614151421417">O&amp;M personnel</span> and send the collected logs.</span></li></ol>
</div>
<div class="section" id="ALM-18003__section1529716184534"><h4 class="sectiontitle">Alarm Clearing</h4><p id="ALM-18003__p4677152685316">After the fault is rectified, the system automatically clears this alarm.</p>
</div>
<div class="section" id="ALM-18003__sdd6932d191f44134b6f3bfc3fe4a218c"><h4 class="sectiontitle">Related Information</h4><p id="ALM-18003__en-us_topic_0070543683_p56935750">None</p>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1298.html">Alarm Reference (Applicable to MRS 3.x)</a></div>
</div>
</div>
View Git Blame Copy Permalink