forked from docs/doc-exports
Yang, Tong
3b1f73dece
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
86 lines
12 KiB
HTML
86 lines
12 KiB
HTML
<a name="ALM-13004"></a><a name="ALM-13004"></a>
|
|
|
|
<h1 class="topictitle1">ALM-13004 ZooKeeper Heap Memory Usage Exceeds the Threshold</h1>
|
|
<div id="body38580247"><div class="section" id="ALM-13004__s960b496ebc144b069eeaef449d9f511c"><h4 class="sectiontitle">Description</h4><p id="ALM-13004__en-us_topic_0070543636_p56530968">The system checks the heap memory usage of the ZooKeeper service every 60 seconds. The alarm is generated when the heap memory usage of a ZooKeeper instance exceeds the threshold (95% of the maximum memory).</p>
|
|
<p id="ALM-13004__en-us_topic_0070543636_p39016666">The alarm is cleared when the memory usage is less than the threshold.</p>
|
|
</div>
|
|
<div class="section" id="ALM-13004__s39176e2253cf4b5ca555b88d806bfb33"><h4 class="sectiontitle">Attribute</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-13004__en-us_topic_0070543636_table6233405" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-13004__en-us_topic_0070543636_row59367431"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.1"><p id="ALM-13004__en-us_topic_0070543636_p44032603">Alarm ID</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.2"><p id="ALM-13004__en-us_topic_0070543636_p9871120">Alarm Severity</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.3"><p id="ALM-13004__en-us_topic_0070543636_p61363278">Auto Clear</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="ALM-13004__en-us_topic_0070543636_row4369596"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.1 "><p id="ALM-13004__en-us_topic_0070543636_p18392962">13004</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.2 "><p id="ALM-13004__en-us_topic_0070543636_p13434953">Major</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.3 "><p id="ALM-13004__en-us_topic_0070543636_p14489400">Yes</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="ALM-13004__s8f55c6176d8b4742a27b3815401d2048"><h4 class="sectiontitle">Parameters</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-13004__en-us_topic_0070543636_table32790776" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-13004__en-us_topic_0070543636_row37629530"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.1"><p id="ALM-13004__en-us_topic_0070543636_p28093062">Name</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.2"><p id="ALM-13004__en-us_topic_0070543636_p60945575">Meaning</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="ALM-13004__row19855645103716"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-13004__p192431315431">Source</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-13004__p692551319435">Specifies the cluster for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="ALM-13004__en-us_topic_0070543636_row37644581"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-13004__en-us_topic_0070543636_p29312231">ServiceName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-13004__en-us_topic_0070543636_p25480534">Specifies the service name for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="ALM-13004__en-us_topic_0070543636_row27998217"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-13004__en-us_topic_0070543636_p53263142">RoleName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-13004__en-us_topic_0070543636_p19347235">Specifies the role name for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="ALM-13004__en-us_topic_0070543636_row39907389"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-13004__en-us_topic_0070543636_p11273070">HostName</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-13004__en-us_topic_0070543636_p40703443">Specifies the object (host ID) for which the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="ALM-13004__en-us_topic_0070543636_row30786668"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-13004__en-us_topic_0070543636_p10692219">Trigger Condition</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-13004__en-us_topic_0070543636_p60763443">Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="ALM-13004__sb1cb768369bf447e88e08c02391abfa5"><h4 class="sectiontitle">Impact on the System</h4><p id="ALM-13004__en-us_topic_0070543636_p22891835">If the available ZooKeeper heap memory is insufficient, a memory overflow occurs and the service breaks down.</p>
|
|
</div>
|
|
<div class="section" id="ALM-13004__s602e19f8d5394ce1b780a76793c5ad6b"><h4 class="sectiontitle">Possible Causes</h4><p id="ALM-13004__en-us_topic_0070543636_p42299323">The heap memory of the ZooKeeper instance is overused or the heap memory is inappropriately allocated.</p>
|
|
</div>
|
|
<div class="section" id="ALM-13004__sdc905a295a9d4f1ca47448ec8692ba34"><h4 class="sectiontitle">Procedure</h4><p class="tableheading" id="ALM-13004__en-us_topic_0070543636_p3693148"><strong id="ALM-13004__b14977785161725">Check heap memory usage.</strong></p>
|
|
<ol id="ALM-13004__ol8396434161734"><li id="ALM-13004__li43146516161727"><span>On the FusionInsight Manager portal, On the displayed interface, click the drop-down button of <strong id="ALM-13004__b1799042311311">ZooKeeper Heap Memory Usage Exceeds the Threshold</strong> and confirm the node IP address of the host for which the alarm is generated in the Location Information.</span></li><li id="ALM-13004__li38538052161727"><span>On the FusionInsight Manager portal, choose <strong id="ALM-13004__b1043145892113">Cluster > </strong><em id="ALM-13004__i1246185811213">Name of the desired cluster</em><strong id="ALM-13004__b144175832120"> > Services</strong> > <strong id="ALM-13004__b5206933161727">ZooKeeper</strong> > <strong id="ALM-13004__b1270010446591">Instance</strong>, click <strong id="ALM-13004__b137401931115915">quorumpeer</strong> in the <strong id="ALM-13004__b1874043112596">Role</strong> column of the corresponding IP address. Click the drop-down menu in the upper right corner of <strong id="ALM-13004__b3273144141318">Chart</strong>, choose <strong id="ALM-13004__b7246166191312">Customize</strong> > <strong id="ALM-13004__b24815175518">CPU and Memory</strong>, and select <strong id="ALM-13004__b540245213164">ZooKeeper Heap And Direct Buffer Resource Percentage</strong>, click <strong id="ALM-13004__b11181773617">OK</strong>. Check the heap memory usage.</span></li><li id="ALM-13004__li15150177161727"><span>Check whether the used heap memory of ZooKeeper reaches 95% of the maximum heap memory specified for ZooKeeper.</span><p><ul class="subitemlist" id="ALM-13004__ul38966055161727"><li id="ALM-13004__li34574471161727">If yes, go to <a href="#ALM-13004__li66283273161727">4</a>.</li><li id="ALM-13004__li49068804161727">If no, go to <a href="#ALM-13004__li34986499161727">7</a>.</li></ul>
|
|
</p></li><li id="ALM-13004__li66283273161727"><a name="ALM-13004__li66283273161727"></a><a name="li66283273161727"></a><span>On the FusionInsight Manager portal, choose <strong id="ALM-13004__b748019632216">Cluster > </strong><em id="ALM-13004__i2482567220">Name of the desired cluster</em><strong id="ALM-13004__b1148019632215"> > Services</strong> > <strong id="ALM-13004__b19204806161727">ZooKeeper</strong> > <strong id="ALM-13004__b38625532161727">Configurations</strong> > <strong id="ALM-13004__b12085468161727">All</strong> <strong id="ALM-13004__b126322010243">Configurations</strong> > <strong id="ALM-13004__b41660353161727">quorumpeer</strong> > <strong id="ALM-13004__b39398858161727">System</strong>. Increase the value of <strong id="ALM-13004__b19045410161727">-Xmx</strong> in <strong id="ALM-13004__b37190969161727">GC_OPTS</strong> as required. The details are as follows:</span><p><ol type="a" id="ALM-13004__ol950916206234"><li id="ALM-13004__li1350932052313">On the <strong id="ALM-13004__b9180153815572">Instance</strong> tab, click <strong id="ALM-13004__b01801138195713">quorumpeer</strong> in the <strong id="ALM-13004__b1318053816574">Role</strong> column of the corresponding IP address. Choose <strong id="ALM-13004__b31801838125713">Customize</strong> > <strong id="ALM-13004__b49761821186">CPU and Memory</strong> in the upper right corner, and select <strong id="ALM-13004__b3180103816578">ZooKeeper Heap And Direct Buffer Resource</strong>, click <strong id="ALM-13004__b1260720351287">OK</strong> to check the heap memory used by ZooKeeper.</li><li id="ALM-13004__li850962017236">Change the value of <strong id="ALM-13004__b395155514577">-Xmx</strong> in the <strong id="ALM-13004__b199518555577">GC_OPTS</strong> parameter based on the actual heap memory usage. Generally, the value is twice the size of the ZooKeeper data volume. For example, if 2 GB ZooKeeper heap memory is used, the following configurations are recommended: -Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=512M -XX:MetaspaceSize=64M -XX:MaxMetaspaceSize=64M -XX:CMSFullGCsBeforeCompaction=1</li></ol>
|
|
</p></li><li id="ALM-13004__li59678545161727"><span>Save the configuration and restart the ZooKeeper service.</span></li><li id="ALM-13004__li43774618161727"><span>Check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-13004__ul37823129161727"><li id="ALM-13004__li235995161727">If yes, no further action is required.</li><li id="ALM-13004__li19115650161727">If no, go to <a href="#ALM-13004__li34986499161727">7</a>.</li></ul>
|
|
</p></li></ol>
|
|
<p class="tableheading" id="ALM-13004__p4863846161727"><strong id="ALM-13004__b60263252161739">Collect fault information.</strong></p>
|
|
<ol start="7" id="ALM-13004__ol51595112161741"><li id="ALM-13004__li34986499161727"><a name="ALM-13004__li34986499161727"></a><a name="li34986499161727"></a><span>On the FusionInsight Manager portal, choose <strong id="ALM-13004__b39977366113627">O&M</strong> > <strong id="ALM-13004__b24251979113627">Log > Download</strong>.</span></li><li id="ALM-13004__li34107122161727"><span>Select <strong id="ALM-13004__b46443037161727">ZooKeeper</strong> in the required cluster from the <strong id="ALM-13004__b15334156161727">Service</strong>.</span></li><li id="ALM-13004__li1145664103113"><span>Click <span><img id="ALM-13004__image1945644173117" src="en-us_image_0269383945.png"></span> in the upper right corner, and set <strong id="ALM-13004__b6456941173117">Start Date</strong> and <strong id="ALM-13004__b11456154113318">End Date</strong> for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click <strong id="ALM-13004__b13456164113319">Download</strong>.</span></li><li id="ALM-13004__li20250462161727"><span>Contact the <span id="ALM-13004__text4614151421417">O&M personnel</span> and send the collected logs.</span></li></ol>
|
|
</div>
|
|
<div class="section" id="ALM-13004__section1529716184534"><h4 class="sectiontitle">Alarm Clearing</h4><p id="ALM-13004__p4677152685316">After the fault is rectified, the system automatically clears this alarm.</p>
|
|
</div>
|
|
<div class="section" id="ALM-13004__sb2eb8883fb1940d0b05b690215576d2e"><h4 class="sectiontitle">Related Information</h4><p id="ALM-13004__en-us_topic_0070543636_p64481034">None</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1298.html">Alarm Reference (Applicable to MRS 3.x)</a></div>
|
|
</div>
|
|
</div>
|
|
|