vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/umn/ALM-19006.html
Yang, Tong 3b1f73dece MRS UMN 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-13 12:03:34 +00:00

109 lines
18 KiB
HTML
Raw Permalink Blame History

<a name="ALM-19006"></a><a name="ALM-19006"></a>
<h1 class="topictitle1">ALM-19006 HBase Replication Sync Failed</h1>
<div id="body19137257"><div class="section" id="ALM-19006__se4824cd5d196465b8d73f2eb2bee6f27"><h4 class="sectiontitle">Description</h4><p id="ALM-19006__en-us_topic_0070543520_p34566005">The alarm module checks the HBase DR data synchronization status every 30 seconds. When disaster recovery (DR) data fails to be synchronized to a standby cluster, the alarm is triggered.</p>
<p id="ALM-19006__en-us_topic_0070543520_p42658596">When DR data synchronization succeeds, the alarm is cleared.</p>
<div class="note" id="ALM-19006__en-us_topic_0070543520_note48383044"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p class="text" id="ALM-19006__en-us_topic_0070543520_p32794215">If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of <strong id="ALM-19006__en-us_topic_0070543520_b26712487">ServiceName</strong> in <strong id="ALM-19006__en-us_topic_0070543520_b39085796">Location</strong>. For example, if the HBase1 service is unavailable, <strong id="ALM-19006__en-us_topic_0070543520_b11832897">ServiceName=HBase1</strong> is displayed in <strong id="ALM-19006__en-us_topic_0070543520_b39387211">Location</strong>, and the operation object in the procedure needs to be changed from HBase to HBase1.</p>
</div></div>
</div>
<div class="section" id="ALM-19006__s6baa8c91d3d941128fb7b103a0e72522"><h4 class="sectiontitle">Attribute</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-19006__en-us_topic_0070543520_table36247516" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-19006__en-us_topic_0070543520_row18100201"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.1"><p id="ALM-19006__en-us_topic_0070543520_p56830139">Alarm ID</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.2"><p id="ALM-19006__en-us_topic_0070543520_p39838562">Alarm Severity</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.2.1.4.1.3"><p id="ALM-19006__en-us_topic_0070543520_p5698118">Automatically Cleared</p>
</th>
</tr>
</thead>
<tbody><tr id="ALM-19006__en-us_topic_0070543520_row58894414"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.1 "><p id="ALM-19006__en-us_topic_0070543520_p5718243">19006</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.2 "><p id="ALM-19006__en-us_topic_0070543520_p60524524">Critical</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.2.1.4.1.3 "><p id="ALM-19006__en-us_topic_0070543520_p3539410">Yes</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="ALM-19006__se44896fecc0247b18c87100c926924ac"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="ALM-19006__en-us_topic_0070543520_table18256764" frame="border" border="1" rules="all"><thead align="left"><tr id="ALM-19006__en-us_topic_0070543520_row6882862"><th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.1"><p id="ALM-19006__en-us_topic_0070543520_p20640979">Name</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.3.1.2"><p id="ALM-19006__en-us_topic_0070543520_p61306625">Meaning</p>
</th>
</tr>
</thead>
<tbody><tr id="ALM-19006__row13141184315104"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-19006__p192431315431">Source</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-19006__p692551319435">Specifies the cluster for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-19006__en-us_topic_0070543520_row66889567"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-19006__en-us_topic_0070543520_p49345813">ServiceName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-19006__en-us_topic_0070543520_p37587916">Specifies the service for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-19006__en-us_topic_0070543520_row2746927"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-19006__en-us_topic_0070543520_p21174538">RoleName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-19006__en-us_topic_0070543520_p37415993">Specifies the role for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-19006__en-us_topic_0070543520_row1199620"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-19006__en-us_topic_0070543520_p30060365">HostName</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-19006__en-us_topic_0070543520_p18970462">Specifies the host for which the alarm is generated.</p>
</td>
</tr>
<tr id="ALM-19006__row055895518017"><td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.1 "><p id="ALM-19006__p26086497">Trigger Condition</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.3.1.2 "><p id="ALM-19006__p14558185516011">Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="ALM-19006__s5887104fc2c449a8824e76eea7339a25"><h4 class="sectiontitle">Impact on the System</h4><p id="ALM-19006__en-us_topic_0070543520_p60212448">HBase data in a cluster fails to be synchronized to the standby cluster, causing data inconsistency between active and standby clusters.</p>
</div>
<div class="section" id="ALM-19006__s8035331f27bb4b889e704fe8d58f8d58"><h4 class="sectiontitle">Possible Causes</h4><ul id="ALM-19006__en-us_topic_0070543520_ul45370106"><li id="ALM-19006__en-us_topic_0070543520_li5677776">The HBase service on the standby cluster is abnormal.</li><li id="ALM-19006__en-us_topic_0070543520_li51099989">A network exception occurs.</li></ul>
</div>
<div class="section" id="ALM-19006__s71895df776534d31b2da3b7303304ca7"><h4 class="sectiontitle">Procedure</h4><p class="tableheading" id="ALM-19006__en-us_topic_0070543520_p45458457"><strong id="ALM-19006__b25525818194110">Observe whether the system automatically clears the alarm.</strong></p>
<ol id="ALM-19006__ol64528750194119"><li id="ALM-19006__li3534916119413"><span>On the FusionInsight Manager portal of the active cluster, click <span class="menucascade" id="ALM-19006__menucascade9622183132519"><b><span class="uicontrol" id="ALM-19006__uicontrol862223115259">O&amp;M</span></b> &gt; <b><span class="uicontrol" id="ALM-19006__uicontrol106221731182512">Alarm</span></b> &gt; <b><span class="uicontrol" id="ALM-19006__uicontrol9622153122518">Alarms.</span></b></span></span></li><li id="ALM-19006__li5777190419413"><span>In the alarm list, click the alarm to obtain alarm generation time from <strong id="ALM-19006__b4970699819413">Generated</strong> of the alarm. Check whether the alarm has existed for five minutes.</span><p><ul class="subitemlist" id="ALM-19006__ul641910019413"><li id="ALM-19006__li6472421119413">If yes, go to <a href="#ALM-19006__li2065263819413">4</a>.</li><li id="ALM-19006__li816977319413">If no, go to <a href="#ALM-19006__li5327925819413">3</a>.</li></ul>
</p></li><li id="ALM-19006__li5327925819413"><a name="ALM-19006__li5327925819413"></a><a name="li5327925819413"></a><span>Wait five minutes and check whether the system automatically clears the alarm.</span><p><ul class="subitemlist" id="ALM-19006__ul1059982219413"><li id="ALM-19006__li5018509019413">If yes, no further action is required.</li><li id="ALM-19006__li3846046019413">If no, go to <a href="#ALM-19006__li2065263819413">4</a>.</li></ul>
</p></li></ol>
<p class="tableheading" id="ALM-19006__p2828953819413"><strong id="ALM-19006__b28700433194131">Check the HBase service status of the standby cluster.</strong></p>
<ol start="4" id="ALM-19006__ol64924621194153"><li id="ALM-19006__li2065263819413"><a name="ALM-19006__li2065263819413"></a><a name="li2065263819413"></a><span>Log in to the FusionInsight Manager portal of the active cluster, and click <span class="menucascade" id="ALM-19006__menucascade11443462617"><b><span class="uicontrol" id="ALM-19006__uicontrol341034192618">O&amp;M</span></b> &gt; <b><span class="uicontrol" id="ALM-19006__uicontrol241434202620">Alarm</span></b> &gt; <b><span class="uicontrol" id="ALM-19006__uicontrol16493412620">Alarms.</span></b></span></span></li><li id="ALM-19006__li916715319413"><span>In the alarm list, click the alarm to obtain <strong id="ALM-19006__b5165602119413">HostName</strong> from <strong id="ALM-19006__b6225101119413">Location</strong>.</span></li><li id="ALM-19006__li1113073719413"><span>Access the node where the HBase client of the active cluster resides as user <strong id="ALM-19006__b1539551419413">omm</strong>.</span><p><p class="litext" id="ALM-19006__p1614982919413">If the cluster uses a security mode, perform security authentication first and then access the <strong id="ALM-19006__b434190319413">hbase shell</strong> interface as user <strong id="ALM-19006__b3907712719413">hbase</strong>.</p>
<p id="ALM-19006__p9416156152318"><strong id="ALM-19006__b1525104553116">cd <span id="ALM-19006__ph381512063917">/opt/client</span></strong></p>
<p class="litext" id="ALM-19006__p81551352143820"><strong id="ALM-19006__b8155185210380">source ./bigdata_env</strong></p>
<p class="litext" id="ALM-19006__p19979739162423"><strong id="ALM-19006__b1925362210265">kinit </strong><em id="ALM-19006__i10521323162611">hbaseuser</em></p>
</p></li><li id="ALM-19006__li4024786019413"><span>Run the <strong id="ALM-19006__b3306777219413">status 'replication', 'source'</strong> command to check the DR synchronization status of the faulty node.</span><p><p class="litext" id="ALM-19006__p2917449519413">The DR synchronization status of a node is as follows.</p>
<pre class="screen" id="ALM-19006__screen1938506519413"><strong id="ALM-19006__b6124386719413">10-10-10-153</strong>:
SOURCE: PeerID=abc, SizeOfLogQueue=0, ShippedBatches=2, ShippedOps=2, ShippedBytes=320, LogReadInBytes=1636, LogEditsRead=5, LogEditsFiltered=3, SizeOfLogToReplicate=0, TimeForLogToReplicate=0, ShippedHFiles=0, SizeOfHFileRefsQueue=0, AgeOfLastShippedOp=0, TimeStampsOfLastShippedOp=Mon Jul 18 09:53:28 CST 2016, Replication Lag=0, FailedReplicationAttempts=0
SOURCE: <strong id="ALM-19006__b1432389819413">PeerID=abc1</strong>, SizeOfLogQueue=0, ShippedBatches=1, ShippedOps=1, ShippedBytes=160, LogReadInBytes=1636, LogEditsRead=5, LogEditsFiltered=3, SizeOfLogToReplicate=0, TimeForLogToReplicate=0, ShippedHFiles=0, SizeOfHFileRefsQueue=0, AgeOfLastShippedOp=16788, TimeStampsOfLastShippedOp=Sat Jul 16 13:19:00 CST 2016, Replication Lag=16788, <strong id="ALM-19006__b6180621919413">FailedReplicationAttempts=5</strong></pre>
</p></li><li id="ALM-19006__li2323816019413"><span>Obtain <strong id="ALM-19006__b2668642119413">PeerID</strong> corresponding to a record whose <strong id="ALM-19006__b3885119719413">FailedReplicationAttempts</strong> value is greater than 0.</span><p><p class="litext" id="ALM-19006__p258201719413">In the preceding step, data on the faulty node 10-10-10-153 fails to be synchronized to a standby cluster whose <strong id="ALM-19006__b1411645319413">PeerID</strong> is <strong id="ALM-19006__b5993921419413">abc1</strong>.</p>
</p></li><li id="ALM-19006__li6555881219413"><a name="ALM-19006__li6555881219413"></a><a name="li6555881219413"></a><span>Run the <strong id="ALM-19006__b781685019413">list_peers</strong> command to find the cluster and the HBase instance corresponding to the <strong id="ALM-19006__b324279419413">PeerID</strong> value.</span><p><pre class="screen" id="ALM-19006__screen1518712219413">PEER_ID CLUSTER_KEY STATE TABLE_CFS
<strong id="ALM-19006__b2918515219413">abc1</strong> 10.10.10.110,10.10.10.119,10.10.10.133:2181:<strong id="ALM-19006__b6133978119413">/hbase2</strong> ENABLED
abc 10.10.10.110,10.10.10.119,10.10.10.133:2181:/hbase ENABLED </pre>
<p class="litext" id="ALM-19006__p2219739319413">In the preceding information, <strong id="ALM-19006__b246637719413">/hbase2</strong> indicates that data is synchronized to the HBase2 instance of the standby cluster.</p>
</p></li><li id="ALM-19006__li5088708919413"><span>In the service list of FusionInsight Manager of the standby cluster, check whether the running status of the HBase instance obtained by using <a href="#ALM-19006__li6555881219413">9</a> is <strong id="ALM-19006__b7892151111569">Normal</strong>.</span><p><ul class="subitemlist" id="ALM-19006__ul565412119413"><li id="ALM-19006__li3066242219413">If yes, go to <a href="#ALM-19006__li2284519319413">14</a>.</li><li id="ALM-19006__li62823519413">If no, go to <a href="#ALM-19006__li448244019413">11</a>.</li></ul>
</p></li><li id="ALM-19006__li448244019413"><a name="ALM-19006__li448244019413"></a><a name="li448244019413"></a><span>In the alarm list, check whether the <strong id="ALM-19006__b1988616223112">ALM-19000 HBase Service Unavailable</strong> alarm is generated.</span><p><ul class="subitemlist" id="ALM-19006__ul2286767019413"><li id="ALM-19006__li5259537519413">If yes, go to <a href="#ALM-19006__li2753337519413">12</a>.</li><li id="ALM-19006__li3236701419413">If no, go to <a href="#ALM-19006__li2284519319413">14</a>.</li></ul>
</p></li><li id="ALM-19006__li2753337519413"><a name="ALM-19006__li2753337519413"></a><a name="li2753337519413"></a><span>Follow troubleshooting procedures in <strong id="ALM-19006__b11821135219302">ALM-19000 HBase Service Unavailable</strong> to rectify the fault.</span></li><li id="ALM-19006__li1519512019413"><span>Wait for a few minutes and check whether the alarm is cleared.</span><p><ul class="subitemlist" id="ALM-19006__ul5652589919413"><li id="ALM-19006__li4647379019413">If yes, no further action is required.</li><li id="ALM-19006__li628065519413">If no, go to <a href="#ALM-19006__li2284519319413">14</a>.</li></ul>
</p></li></ol>
<p class="tableheading" id="ALM-19006__p3897104819413"><strong id="ALM-19006__b55853063194210">Check network connections between RegionServers on active and standby clusters.</strong></p>
<ol start="14" id="ALM-19006__ol41239844194223"><li id="ALM-19006__li2284519319413"><a name="ALM-19006__li2284519319413"></a><a name="li2284519319413"></a><span>Log in to the FusionInsight Manager portal of the active cluster, and click <span class="menucascade" id="ALM-19006__menucascade45089526261"><b><span class="uicontrol" id="ALM-19006__uicontrol19508195219261">O&amp;M</span></b> &gt; <b><span class="uicontrol" id="ALM-19006__uicontrol175081752112617">Alarm</span></b> &gt; <b><span class="uicontrol" id="ALM-19006__uicontrol1250845262617">Alarms.</span></b></span></span></li><li id="ALM-19006__li3322104919413"><a name="ALM-19006__li3322104919413"></a><a name="li3322104919413"></a><span>In the alarm list, click the alarm to obtain <strong id="ALM-19006__b428014919413">HostName</strong> from <strong id="ALM-19006__b3852134319413">Location</strong>.</span></li><li id="ALM-19006__li5895388719413"><span>Use the IP address obtained in <a href="#ALM-19006__li3322104919413">15</a> to log in to a faulty RegionServer node as user <strong id="ALM-19006__b3055398719413">omm</strong>.</span></li><li id="ALM-19006__li1392399119413"><span>Run the <strong id="ALM-19006__b6082293919413">ping</strong> command to check whether network connections between the faulty RegionServer node and the host where RegionServer of the standby cluster resides are in the normal state.</span><p><ul class="subitemlist" id="ALM-19006__ul154711019413"><li id="ALM-19006__li2771099419413">If yes, go to <a href="#ALM-19006__li342888619413">20</a>.</li><li id="ALM-19006__li2999806219413">If no, go to <a href="#ALM-19006__li5820706019413">18</a>.</li></ul>
</p></li><li id="ALM-19006__li5820706019413"><a name="ALM-19006__li5820706019413"></a><a name="li5820706019413"></a><span>Contact the network administrator to restore the network.</span></li><li id="ALM-19006__li5579067219413"><span>After the network is running properly, check whether the alarm is cleared in the alarm list.</span><p><ul class="subitemlist" id="ALM-19006__ul4708502519413"><li id="ALM-19006__li5410149219413">If yes, no further action is required.</li><li id="ALM-19006__li2014475019413">If no, go to <a href="#ALM-19006__li342888619413">20</a>.</li></ul>
</p></li></ol>
<p class="tableheading" id="ALM-19006__p2111204419413"><strong id="ALM-19006__b15875429194233">Collect fault information.</strong></p>
<ol start="20" id="ALM-19006__ol17286287194237"><li id="ALM-19006__li342888619413"><a name="ALM-19006__li342888619413"></a><a name="li342888619413"></a><span>On the FusionInsight Manager interface of active and standby clusters, choose <strong id="ALM-19006__b87211374515">O&amp;M</strong> &gt; <strong id="ALM-19006__b1558615387574">Log </strong>&gt;<strong id="ALM-19006__b155871538105719"> Download</strong>.</span></li><li id="ALM-19006__li1545387419413"><span>In the <strong id="ALM-19006__b3085997719413">Service</strong> drop-down list box, select <strong id="ALM-19006__b18463135412711">HBase </strong>in the required cluster.</span></li><li id="ALM-19006__li1145664103113"><span>Click <span><img id="ALM-19006__image1945644173117" src="en-us_image_0269417420.png"></span> in the upper right corner, and set <strong id="ALM-19006__b6456941173117">Start Date</strong> and <strong id="ALM-19006__b11456154113318">End Date</strong> for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click <strong id="ALM-19006__b13456164113319">Download</strong>.</span></li><li id="ALM-19006__li3956018919413"><span>Contact the <span id="ALM-19006__text4614151421417">O&amp;M personnel</span> and send the collected fault logs.</span></li></ol>
</div>
<div class="section" id="ALM-19006__section1529716184534"><h4 class="sectiontitle">Alarm Clearing</h4><p id="ALM-19006__p4677152685316">After the fault is rectified, the system automatically clears this alarm.</p>
</div>
<div class="section" id="ALM-19006__s3ecfd41e58d645de9a90fdb22deb2672"><h4 class="sectiontitle">Related Information</h4><p id="ALM-19006__en-us_topic_0070543520_p963478">None</p>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1298.html">Alarm Reference (Applicable to MRS 3.x)</a></div>
</div>
</div>
View Git Blame Copy Permalink