ALM-38006 Percentage of Kafka Partitions That Are Not Completely Synchronized Exceeds the Threshold

Description

The system checks the percentage of Kafka partitions that are not completely synchronized to the total number of partitions every 60 seconds. This alarm is generated when the percentage exceeds the threshold (50% by default) for 3 consecutive times.

When the Trigger Count is 1, this alarm is cleared when the percentage is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the percentage is less than or equal to 90% of the threshold.

Attribute

Parameters

Impact on the System

Too many Kafka partitions that are not completely synchronized affect service reliability. In addition, data may be lost when leaders are switched.

Possible Causes

Some nodes where the Broker instance resides are abnormal or stop running. As a result, replicas of some partitions in Kafka are out of the in-sync replicas (ISR) set.

Procedure

Check Broker instances.

1. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Kafka > Instance. The Kafka instances page is displayed.
2. Check whether faulty nodes exist among all Broker nodes.
   • If yes, record the host name of the node and go to 3.
   • If no, go to 5.

3. On the FusionInsight Manager portal, click O&M > Alarm > Alarms to check whether the fault described in 2 exists in the alarm information and handle the alarm based on corresponding methods.
4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Kafka > Instance. The Kafka instances page is displayed.
5. Check whether stopped nodes exist among all Broker instance.
   • If yes, go to 6.
   • If no, go to 7.

6. Select all stopped Broker instances and click Start Instance.
7. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 8.

Collect fault information.

8. On the FusionInsight Manager portal, choose O&M > Log > Download.
9. Select Kafka in the required cluster from the Service drop-down list.
10. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
11. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None