vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/umn/ALM-12045.html
Yang, Tong 2195db241c MRS UMN 20231220 version update 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Reviewed-by: Rechenburg, Matthias <matthias.rechenburg@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2024-05-16 09:40:21 +00:00

25 KiB
Raw Blame History

ALM-12045 Read Packet Dropped Rate Exceeds the Threshold

Description

The system checks the read packet dropped rate every 30 seconds. This alarm is generated when the read packet dropped rate exceeds the threshold (the default threshold is 0.5%) for multiple times (the default value is 5).

To change the threshold, choose O&M > Alarm > Thresholds > Name of the desired cluster > Host > Network Reading > Read Packet Dropped Rate.

This alarm is cleared when Trigger Count is 1 and the read packet dropped rate is less than or equal to the threshold. This alarm is cleared when Trigger Count is greater than 1 and the read packet dropped rate is less than or equal to 90% of the threshold.

The alarm detection is disabled by default. If you want to enable this function, check whether this function can be enabled based on Checking System Environments.

Attribute

Alarm ID

Alarm Severity

Auto Clear

12045

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster or system for which the alarm is generated.

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

PortName

Specifies the network port for which the alarm is generated.

Trigger Condition

Specifies the threshold for triggering the alarm.

Impact on the System

The service performance deteriorates or some services time out.

Risk warning: In SUSE kernel 3.0 or later or Red Hat 7.2, the system kernel modifies the mechanism for counting the number of dropped read packets. In this case, this alarm may be generated even if the network is running properly, but services are not affected. You are advised to check the system environment first.

Possible Causes

  • An OS exception occurs.
  • The NICs are bonded in active/standby mode.
  • The alarm threshold is improperly configured.
  • The network quality is poor.

Procedure

View the network packet dropped rate.

  1. On MRS Manager, choose O&M > Alarm > Alarms. On the page that is displayed, click in the row containing the alarm, and view the name of the host for which the alarm is generated and the NIC name.
  2. Log in to the alarm node as user omm, and run the /sbin/ifconfig NIC name command to check whether packet loss occurs on the network.

    • IP address of the node for which the alarm is generated: Query the IP address of the node for which the alarm is generated on the Hosts page of MRS Manager based on the value of HostName in the alarm location information. Check both the IP addresses of the management plane and service plane.
    • Packet loss rate = (Number of dropped packets/Total number of received packets) x 100%. If the packet loss rate is greater than the system threshold (0.5% by default), read packets are dropped.
    • If yes, go to 11.
    • If no, go to 3.

Check the system environment.

  1. Log in to the active OMS node or the alarm node as user omm.
  2. Run the cat /etc/*-release command to check the OS type.

    • For Red Hat Enterprise Linux, go to 5.
      # cat /etc/*-release
Red Hat Enterprise Linux Server release 7.2 (Santiago)
    • For SUSE Linux, go to 6.
      # cat /etc/*-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3
    • For other OS types, go to 11.

  3. Run the cat /etc/redhat-release command to check whether the OS version is Red Hat 7.2 (x86) or Red Hat 7.4 (TaiShan).

    # cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Santiago)
    • If yes, the alarm sending function cannot be enabled. Go to 7.
    • If no, go to 11.

  4. Run the cat /proc/version command to check whether the SUSE kernel version is 3.0 or later.

    # cat /proc/version
Linux version 3.0.101-63-default (geeko@buildhost) (gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Tue Jun 23 16:02:31 UTC 2015 (4b89d0c)
    • If yes, the alarm sending function cannot be enabled. Go to 7.
    • If no, go to 11.

  5. Log in to MRS Manager and choose O&M > Alarm > Threshold Configuration.
  1. In the navigation tree of the Thresholds page, choose Name of the desired cluster > Host > Network Reading > Read Packet Dropped Rate. In the area on the right, check whether the Switch is toggled on.

    • If yes, the alarm sending function is enabled. Go to 9.
    • If no, the alarm sending function is disabled. Go to 10.

  2. In the area on the right, toggle Switch off to disable the checking of Network Read Packet Dropped Rate Exceeds the Threshold.

  3. On the Alarm page of MRS Manager, search for alarm 12045 and manually clear the alarm if it is not automatically cleared. No further action is required.

    ID of the Network Read Packet Dropped Rate Exceeds the Threshold alarm is 12045.

Check whether the NICs are bonded in active/standby mode.

  1. Log in to the alarm node as user omm and run the ls -l /proc/net/bonding command to check whether the /proc/net/bonding directory exists on the node.

    • If yes, the bond mode is configured for the node. Go to 12.
      # ls -l /proc/net/bonding/
total 0
-r--r--r-- 1 root root 0 Oct 11 17:35 bond0
    • If no, the bond mode is not configured for the node. Go to 14.
      # ls -l /proc/net/bonding/
ls: cannot access /proc/net/bonding/: No such file or directory

  2. Run the cat /proc/net/bonding/bond0 command to check whether the value of Bonding Mode in the configuration file is fault-tolerance.

    In the command, bond0 indicates the name of the bond configuration file. Use the file name obtained in 11.

    # cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth1 (primary_reselect always)
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 1
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 1
Slave queue ID: 0
    • If yes, the NICs are bonded in active/standby mode. Go to 13.
    • If no, go to 14.

  3. Check whether the NIC specified by NetworkCardName in the alarm is the standby NIC.

    • If yes, the alarm of the standby NIC cannot be automatically cleared. Manually clear the alarm on the alarm management page. No further action is required.
    • If no, go to 14.

      To determine the standby NIC, check the /proc/net/bonding/bond0 configuration file. If the NIC name corresponding to NetworkCardName is Slave Interface but not Currently Active Slave (the current active NIC), the NIC is the standby one.

Check whether the threshold is set properly.

  1. Log in to MRS Manager, choose O&M > Alarm > Thresholds > Name of the desired cluster > Host > Network Reading > Read Packet Dropped Rate, and check whether the alarm threshold is configured properly. The default value is 0.5%. You can adjust the threshold as needed.

    • If yes, go to 17.
    • If no, go to 15.

  1. Choose O&M > Alarm > Thresholds > Name of the desired cluster > Host > Network Reading > Read Packet Dropped Rate. Click Modify in the Operation column to change the threshold. See Figure 1.

    Figure 1 Configuring the alarm threshold

  2. After 5 minutes, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 17.

Check whether the network connection is normal.

  1. Contact the network administrator to check whether the network is normal.

    • If yes, rectify the fault and go to 18.
    • If no, go to 19.

  2. After 5 minutes, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 19.

Collect the fault information.

  1. On MRS Manager of the active cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
  2. Select OMS for Service and click OK.
  3. Expand the Hosts dialog box and select the alarm node and the active OMS node.
  4. Click in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.
  5. Contact O&M personnel and provide the collected logs.

Alarm Clearing

This alarm is automatically cleared after the fault is rectified.

Related Information

None