in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.Collect fault information.
- in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Services on the node are unavailable.
The network is disconnected, the hardware is faulty, or the operating system runs slowly.
+Check whether the network is disconnected, whether the hardware is faulty, or whether the operating system runs commands slowly.
-
in the row containing the alarm, click the host name, and view the IP address of the host for which the alarm is generated.
in the row containing the alarm, click the host name, and view the IP address of the host for which the alarm is generated.
If the faulty node is the active management node and fails login, the network of the active management node may be faulty. In this case, go to 4.
+Log in to any management node and view the /var/log/Bigdata/omm/oms/ha/scriptlog/floatip.log log to check whether the logs generated one to two minutes before and after the faults occur are complete.
For example, a complete log is in the following format:
-2017-12-09 04:10:51,000 INFO (floatip) Read from ${BIGDATA_HOME}/om-server_8.1.0.1/om/etc/om/routeSetConf.ini,value is : yes
+2017-12-09 04:10:51,000 INFO (floatip) Read from ${BIGDATA_HOME}/om-server_*/om/etc/om/routeSetConf.ini,value is : yes
2017-12-09 04:10:51,000 INFO (floatip) check wsNetExport : eth0 is up.
2017-12-09 04:10:51,000 INFO (floatip) check omNetExport : eth0 is up.
2017-12-09 04:10:51,000 INFO (floatip) check wsInterface : eRth0:oms, wsFloatIp: XXX.XXX.XXX.XXX.
2017-12-09 04:10:51,000 INFO (floatip) check omInterface : eth0:oms, omFloatIp: XXX.XXX.XXX.XXX.
2017-12-09 04:10:51,000 INFO (floatip) check wsFloatIp : XXX.XXX.XXX.XXX is reachable.
2017-12-09 04:10:52,000 INFO (floatip) check omFloatIp : XXX.XXX.XXX.XXX is reachable.
-
-sssd restart example
Feb 7 11:38:16 10-132-190-105 sssd[pam]: Shutting down Feb 7 11:38:16 10-132-190-105 sssd[nss]: Shutting down @@ -89,11 +91,15 @@ Feb 7 11:38:16 10-132-190-105 sssd[pam]: Starting up
Feb 11 11:44:42 10-120-205-33 nscd: nss_ldap: failed to bind to LDAP server ldaps://10.120.205.55:21780: Can't contact LDAP server Feb 11 11:44:43 10-120-205-33 ntpq: nss_ldap: failed to bind to LDAP server ldaps://10.120.205.55:21780: Can't contact LDAP server Feb 11 11:44:44 10-120-205-33 ntpq: nss_ldap: failed to bind to LDAP server ldaps://10.120.205.92:21780: Can't contact LDAP server- +
Collect the fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.Check whether the memory of the NodeAgent process is insufficient.
+vi /var/log/Bigdata/nodeagent/scriptlog/agent_gc.log.*.current
+Collect fault information.
+
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.This alarm is automatically cleared after the fault is rectified.
Check whether the instance process is abnormal.
-
in the row where the alarm is located , and click the host name to view the host address for which the alarm is generated
in the row where the alarm is located , and click the host name to view the host address for which the alarm is generatedFor example, the NameNode installation directory is ${BIGDATA_HOME}/FusionInsight_Current/1_8_NameNode/etc.
chmod 750 <folder_name>
@@ -75,7 +75,7 @@Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Check whether the network between the active and standby Manager server is normal.
-
in the row containing the alarm and view the IP address of the standby Manager (Peer Manager) server in the alarm details.
in the row containing the alarm and view the IP address of the standby Manager (Peer Manager) server in the alarm details.cd /opt
@@ -77,7 +77,7 @@Collect fault information.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Check whether the network between the active Manager server and the standby Manager server is normal.
-
in the row where the alarm is located and obtain the standby Manager server IP address (Peer Manager IP address) in the alarm details.
in the row where the alarm is located and obtain the standby Manager server IP address (Peer Manager IP address) in the alarm details.vi ha.log.2021-03-22_12-00-07
Check whether error information is reported before and after the alarm generation time.
Collect fault information.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
The system checks whether the NTP service on a node synchronizes time with the NTP service on the active OMS node every 60 seconds. This alarm is generated when the NTP service fails to synchronize time for two consecutive times.
+This alarm is generated when the time difference between the NTP service on a node and the NTP service on the active OMS node is greater than or equal to 20s for two consecutive times. This alarm is cleared when the time difference is less than 20s.
+Alarm ID + |
+Alarm Severity + |
+Auto Clear + |
+
|---|---|---|
12012 + |
+Major + |
+Yes + |
+
Name + |
+Meaning + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
The time on the node is inconsistent with that on other nodes in the cluster. Therefore, some FusionInsight applications on the node may not run properly.
+Check the NTP service mode of the node.
+sh ${BIGDATA_HOME}/om-server/om/sbin/status-oms.sh
+ If both "chrony" and "ntp" are displayed in the ResName column of the command output, the NTP service mode is being switched. Wait for 10 minutes and go to 1 again. If both "chrony" and "ntp" persist, contact O&M personnel personnel.
+Check whether the chrony service on the node is started properly.
+
in the row containing the alarm, and view the name of the host for which the alarm is generated in Location.Check whether the current node can synchronize time properly with the chrony service on the active OMS node.
+Log in to the node for which the alarm is generated as user root and run the chronyc sources command.
+In the command output, if there is an asterisk (*) before the IP address of the chrony service on the active OMS node, the synchronization is normal. The command output is as follows:
+MS Name/IP address Stratum Poll Reach LastRx Last sample +=============================================================================== +^* 10.10.10.162 10 10 377 626 +16us[ +15us] +/- 308us+
In the command output, if there is no asterisk (*) before the IP address of the NTP service on the active OMS node, and the value of Reach is 0, the synchronization is abnormal.
+MS Name/IP address Stratum Poll Reach LastRx Last sample +=============================================================================== +^? 10.1.1.1 0 10 0 - +0ns[ +0ns] +/- 0ns+ +
In Red Hat Enterprise Linux, run the cat ${BIGDATA_HOME}/om-server/OMS/workspace/conf/chrony.keys command.
+In Red Hat Enterprise Linux, run the diff ${BIGDATA_HOME}/om-server/OMS/workspace/conf/chrony.keys /etc/chrony.keys command.
+ If the key values are the same, no result is returned after the command is executed. For example:
+
host01:~ # cat ${BIGDATA_HOME}/om-server/OMS/workspace/conf/chrony.keys
+1 M sdYbq;o^CzEAWo<U=Tw5
+host01:~ # diff ${BIGDATA_HOME}/om-server/OMS/workspace/conf/chrony.keys /etc/chrony.keys
+host01:~ #
+cd ${BIGDATA_HOME}/om-server/OMS/workspace/conf
+sed -i "`cat chrony.keys | grep -n '1 M'|awk -F ':' '{print $1}'`d" chrony.keys
+echo "1 M `cat ${BIGDATA_HOME}/om-server/om/packaged-distributables/ntpKeyFile`" >> chrony.keys
+Check whether the key value of the authentication key index field 1M in chrony.keys is the same as that of ntpKeyFile.
+ +Check whether the time deviation between the node and the chrony service on the active OMS node is large.
+If the time on the alarm node is later than that on the chrony service of the active OMS node, adjust the time of the alarm node. After adjusting the time, choose More > Start All Instances to start the services on the node.
+If the time on the alarm node is earlier than that on the chrony service of the active OMS node, wait until the time deviation is due and adjust the time of the alarm node. After adjusting the time, choose More > Start All Instances to start the services on the node.
+ If you do not wait, data loss may occur.
+Check whether the NTP service on the node is started properly.
+
in the row containing the alarm, and view the name of the host for which the alarm is generated in Location.Check whether the node can synchronize time properly with the NTP service on the active OMS node.
+Log in to the alarm node as user root and run the ntpq -np command.
+If an asterisk (*) exists before the IP address of the NTP service on the active OMS node in the command output, the synchronization is in normal state. The command output is as follows:
+remote refid st t when poll reach delay offset jitter +============================================================================== +*10.10.10.162 .LOCL. 1 u 1 16 377 0.270 -1.562 0.014+
If there is no asterisk (*) before the IP address of the NTP service on the active OMS node, as shown in the following command output, and the value of refid is .INIT., the synchronization is abnormal.
+remote refid st t when poll reach delay offset jitter +============================================================================== +10.10.10.162 .INIT. 1 u 1 16 377 0.270 -1.562 0.014+ +
In SUSE Linux, run the cat ${BIGDATA_HOME}/om-server/OMS/workspace/conf/ntp.keys command.
+In Red Hat Enterprise Linux or EulerOS, run the cat ${BIGDATA_HOME}/om-server/OMS/workspace/conf/ntpkeys command.
+In SUSE Linux, run the diff ${BIGDATA_HOME}/om-server/OMS/workspace/conf/ntp.keys /etc/ntp.keys command.
+In Red Hat Enterprise Linux or EulerOS, run the diff ${BIGDATA_HOME}/om-server/OMS/workspace/conf/ntpkeys /etc/ntp/ntpkeys command.
+ If the key values are the same, no result is returned after the command is executed. For example:
+host01:~ # cat ${BIGDATA_HOME}/om-server/OMS/workspace/conf/ntp.keys
+1 M sdYbq;o^CzEAWo<U=Tw5
+host01:~ # diff ${BIGDATA_HOME}/om-server/OMS/workspace/conf/ntp.keys /etc/ntp.keys
+host01:~ #
+cd ${BIGDATA_HOME}/om-server/OMS/workspace/conf
+sed -i "`cat ntp.keys | grep -n '1 M'|awk -F ':' '{print $1}'`d" ntp.keys
+echo "1 M `cat ${BIGDATA_HOME}/om-server/om/packaged-distributables/ntpKeyFile`" >>ntp.keys
+Check whether the key value of the authentication key index field 1M in ntp.keys is the same as that of ntpKeyFile.
+ +Check whether the time deviation between the node and the NTP service on the active OMS node is large.
+If the time on the alarm node is later than that on the NTP service of the active OMS node, adjust the time of the alarm node. After adjusting the time, choose More > Start All Instances to start the services on the node.
+If the time on the alarm node is earlier than that on the NTP service of the active OMS node, wait until the time deviation is due and adjust the time of the alarm node. After adjusting the time, choose More > Start All Instances to start the services on the node.
+ If you do not wait, data loss may occur.
+Collect the fault information.
+ in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.This alarm is automatically cleared after the fault is rectified.
+None
+
in the row where the alarm is located.
in the row where the alarm is located.The hard disk is faulty, for example, a bad sector exists.
in the row where the alarm is located.
in the row where the alarm is located.After the fault is rectified, the system automatically clears this alarm.
On FusionInsight Manager, choose O&M > Alarm > Thresholds > Name of the desired cluster > Host > CPU > Host CPU Usage and change the alarm smoothing times based on CPU usage, as shown in Figure 1.
This option defines the alarm check phase. Trigger Count indicates the alarm check threshold. An alarm is generated when the number of check times exceeds the threshold.
On Host CPU Usage page and click Modify in the Operation column to change the alarm threshold, as shown in Figure 2.
- +Check whether the CPU usage reaches the upper limit.
-
in the row where the alarm is located to view the alarm host address in the alarm details.
in the row where the alarm is located to view the alarm host address in the alarm details.Collect fault information.
Check whether the alarm threshold is appropriate.
Check whether the disk usage reaches the upper limit.
-
in the row where the alarm is located to view the alarm host name and disk partition information in the alarm details.
in the row where the alarm is located to view the alarm host name and disk partition information in the alarm details.Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Expand the system.
-
in the row where the alarm is located to view the alarm host address in the alarm details.
in the row where the alarm is located to view the alarm host address in the alarm details.Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Too many processes are running on the node. You need to increase the value of pid_max.
Increase the value of pid_max.
-
in the row where the alarm is located to view the alarm host address in the alarm details.Example: echo 65536 > /proc/sys/kernel/pid_max
+
in the row where the alarm is located to view the alarm host address in the alarm details.Example: echo 65536 > /proc/sys/kernel/pid_max
The maximum value of pid_max is as follows:
Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
The system checks the number of processes in the D state of user omm on the host every 30 seconds and compares the actual number with the threshold. The number of processes in the D state on the host has a default threshold range. This alarm is generated when the number of processes exceeds the threshold.
-This alarm is cleared when the Trigger Count is 1 and the total number of processes in the D state of user omm on the host does not exceed the threshold. This alarm is cleared when the Trigger Count is greater than 1 and the total number of processes in the D state of user omm on the host is less than or equal to 90% of the threshold.
+The system checks the number of processes in the D stateand Z state of user omm on the host every 30 seconds and compares the actual number with the threshold. The number of processes in the D state and Z state on the host has a default threshold range. This alarm is generated when the number of processes exceeds the threshold.
+This alarm is cleared when the Trigger Count is 1 and the total number of processes in the D state and Z state of user omm on the host does not exceed the threshold. This alarm is cleared when the Trigger Count is greater than 1 and the total number of processes in the D state and Z state of user omm on the host is less than or equal to 90% of the threshold.
+ The function of checking the number of processes in the Z state on the host applies to MRS 3.2.0 or later.
+Alarm ID @@ -64,13 +66,13 @@Possible CausesThe host responds slowly to I/O (disk I/O and network I/O) requests and some processes are in the D state and Z state. ProcedureCheck the processes in the D state. -
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
12033 |
-Major + | Minor |
Yes |
@@ -81,7 +84,7 @@
Alarm ID + |
+Alarm Severity + |
+Auto Clear + |
+
|---|---|---|
12037 + |
+Major + |
+Yes + |
+
Name + |
+Meaning + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the IP address of the NTP server for which the alarm is generated. + |
+
The NTP server configured on the active OMS node is abnormal. In this case, the active OMS node cannot synchronize time with the NTP server and a time offset may be generated in the cluster.
+Check the NTP server network.
+
in the row where the alarm is located.Check whether the NTP server authentication fails.
+su - omm
+sh ${BIGDATA_HOME}/om-server/om/sbin/status-oms.sh
+ If both "chrony" and "ntp" are displayed in the ResName column of the command output, the NTP service mode is being switched. Wait for 10 minutes and perform 5 again. If both "chrony" and "ntp" still exist in the ResName column, contact O&M personnel.
+If the value of Reach for chrony is 0, the connection or authentication fails.
+ +If refid of the NTP server is .AUTH., the authentication fails.
+ +Check whether the time can be obtained from the NTP server.
+Check whether the time obtained from the NTP server is not continuously updated.
+Collect fault information.
+
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
+None
+Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Check whether the network between the active and standby nodes is normal.
-
in the row where the alarm is located, and query the standby OMS Database IP address.
in the row where the alarm is located, and query the standby OMS Database IP address.Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
The system checks the entropy for five consecutive times at 00:00 every day. Specifically, the system checks whether rng-tools or haveged has been enabled and correctly configured. If neither is configured, the system continues to check the entropy. If the entropy is less than 100 for five consecutive times, this alarm is reported.
+MRS 3.2.0 or later:
+The system checks whether the rng-tools or haveged tool has been enabled and correctly configured every 5 minutes. If neither tool is configured, this alarm is generated. If either is configured, the system continues to check the entropy. If the entropy is less than 100 for five consecutive times, this alarm is generated.
+This alarm is cleared when rng-tools or haveged has been installed and enabled on the target node and the entropy of the OS is greater than or equal to 100 in at least one of five entropy checks.
+MRS 3.1.2 or earlier:
+The system checks the entropy for five consecutive times at 00:00 every day. Specifically, the system checks whether rng-tools or haveged has been enabled and correctly configured. If neither is configured, the system continues to check the entropy. If the entropy is less than 100 for five consecutive times, this alarm is reported.
This alarm is cleared when the system detects that the true random number mode has been configured, the random number parameters have been configured in the pseudo-random number mode, or neither mode is configured but the entropy of the OS is greater than or equal to 100 in at least one of five entropy checks.
Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.This alarm is automatically cleared after the fault is rectified.
If the disk partition where the cluster installation directory resides is used up, some temporary files will be generated in the program installation directory when running the sed command fails. Users do not have the read, write, and execute permissions of these temporary files. The system reports an alarm indicating that permissions of temporary files are abnormal if these files are within the monitoring range of the alarm. Perform the preceding alarm handling processes to clear the alarm. Alternatively, you can directly delete the temporary files after confirming that files with abnormal permissions are temporary. The temporary file generated after a sed command execution failure is similar to the following.
Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
View the network packet dropped rate.
-
in the row containing the alarm, and view the name of the host for which the alarm is generated and the NIC name.
in the row containing the alarm, and view the name of the host for which the alarm is generated and the NIC name.
ID of the Network Read Packet Dropped Rate Exceeds the Threshold alarm is 12045.
Check whether the threshold is set properly.
Check whether the network connection is normal.
@@ -146,7 +146,7 @@ Slave queue ID: 0Collect the fault information.
- in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download. in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.This alarm is automatically cleared after the fault is rectified.
Check whether the threshold is set properly.
See Figure 1.
- +Check whether the network connection is normal.
@@ -81,7 +81,7 @@Collect the fault information.
- in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download. in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.This alarm is automatically cleared after the fault is rectified.
Check whether the threshold is set properly.
See Figure 1.
- +Check whether the network connection is normal.
@@ -81,7 +81,7 @@Collect the fault information.
- in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download. in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.This alarm is automatically cleared after the fault is rectified.
Check whether the threshold is set properly.
See Figure 1.
- +Check whether the network connection is normal.
@@ -81,7 +81,7 @@Collect the fault information.
- in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download. in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.This alarm is automatically cleared after the fault is rectified.
Check whether the threshold is set properly.
For details, see Figure 1.
- +Check whether the network port rate can meet the service requirements.
-
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host and the network port name for which the alarm is generated. In the VM environment, you cannot run a command to query the network port rate. It is recommended that you contact the system administrator to confirm whether the network port rate meets the requirements.
+
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host and the network port name for which the alarm is generated. In the VM environment, you cannot run a command to query the network port rate. It is recommended that you contact the system administrator to confirm whether the network port rate meets the requirements.
Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Check whether the threshold is set properly.
For details, see Figure 1.
- +Check whether the network port rate can meet the service requirements.
-
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host and the network port name for which the alarm is generated. In the VM environment, you cannot run a command to query the network port rate. It is recommended that you contact the system administrator to confirm whether the network port rate meets the requirements.
+
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host and the network port name for which the alarm is generated. In the VM environment, you cannot run a command to query the network port rate. It is recommended that you contact the system administrator to confirm whether the network port rate meets the requirements.
Collect fault information.
-
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.After the fault is rectified, the system automatically clears this alarm.
Massive small files are stored in the disk.
Massive small files are stored in the disk.
-
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host and the disk partition for which the alarm is generated.# df -i | grep -iE "xvda2|Filesystem" +
- On FusionInsight Manager, choose O&M > Alarm > Alarms and click
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host and the disk partition for which the alarm is generated.
- Log in to the host for which the alarm is generated as user root.
- Run the df -i | grep -iE "partition name|Filesystem" command to check the current disk Inode usage.
# df -i | grep -iE "xvda2|Filesystem" Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda2 2359296 207420 2151876 9% /- If the Inode usage exceeds the threshold, manually check small files stored in the disk partition and confirm whether these small files can be deleted.
Run the for i in /*; do echo $i; find $i|wc -l; done command to query the number of files in a partition. Replace /* with the specified partition.
@@ -89,7 +89,7 @@ Filesystem Inodes IUsed IFree IUse% Mounted on- Wait for 5 minutes, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
-+
- On the FusionInsight Manager home page of the active cluster, choose O&M > Log > Download.
- Select OMS from the Service and click OK.
- Set Host to the node for which the alarm is generated and the active OMS node.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On the FusionInsight Manager home page of the active cluster, choose O&M > Log > Download.
- Select OMS from the Service and click OK.
- Set Host to the node for which the alarm is generated and the active OMS node.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12052.html b/docs/mrs/umn/ALM-12052.html index af5544ff..8d6c105f 100644 --- a/docs/mrs/umn/ALM-12052.html +++ b/docs/mrs/umn/ALM-12052.html @@ -66,7 +66,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Possible Causes
- The temporary port cannot meet the current service requirements.
- The system is abnormal.
Procedure
Expand the temporary port number range.
-
- On FusionInsight Manager, click
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host for which the alarm is generated.
- Log in to the host for which the alarm is generated as user omm.
- Run the cat /proc/sys/net/ipv4/ip_local_port_range |cut -f 1 command to obtain the value of the start port and run the cat /proc/sys/net/ipv4/ip_local_port_range |cut -f 2 command to obtain the value of the end port. The total number of temporary ports is the value of the end port minus the value of the start port. If the total number of temporary ports is smaller than 28,232, the random port range of the OS is narrow. Contact the system administrator to increase the port range.
- Run the ss -ant 2>/dev/null | grep -v LISTEN | awk 'NR > 2 {print $4}'|cut -d ':' -f 2 | awk '$1 >"Value of the start port" {print $1}' | sort -u | wc -l command to calculate the number of used temporary ports.
- The formula for calculating the usage of the temporary ports is: Usage of the temporary ports = (Number of used temporary ports/Total number of temporary ports) x 100%. Check whether the temporary port usage exceeds the threshold. +
- On FusionInsight Manager, click
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host for which the alarm is generated.
- Log in to the host for which the alarm is generated as user omm.
- Run the cat /proc/sys/net/ipv4/ip_local_port_range |cut -f 1 command to obtain the value of the start port and run the cat /proc/sys/net/ipv4/ip_local_port_range |cut -f 2 command to obtain the value of the end port. The total number of temporary ports is the value of the end port minus the value of the start port. If the total number of temporary ports is smaller than 28,232, the random port range of the OS is narrow. Contact the system administrator to increase the port range.
- Run the ss -ant 2>/dev/null | grep -v LISTEN | awk 'NR > 2 {print $4}'|cut -d ':' -f 2 | awk '$1 >"Value of the start port" {print $1}' | sort -u | wc -l command to calculate the number of used temporary ports.
- The formula for calculating the usage of the temporary ports is: Usage of the temporary ports = (Number of used temporary ports/Total number of temporary ports) x 100%. Check whether the temporary port usage exceeds the threshold.
- Wait for 5 minutes, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Check whether the system environment is abnormal.
@@ -86,7 +86,7 @@ tcp 0 0 10-120-85-154:45435 10-120-85-154:9866 CLOSE_WAIT 94237/java- After obtaining the administrator's approval, clear the processes that occupy a large number of ports. Wait for 5 minutes, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 10.
Collect fault information.
-+
- On the FusionInsight Manager home page of the active cluster, choose O&M > Log > Download.
- Select OMS from the Service and click OK.
- Set Host to the node for which the alarm is generated and the active OMS node.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information and files port_result.txt and ps_result.txt. Then, delete the two residual temporary files from the environment.
- On the FusionInsight Manager home page of the active cluster, choose O&M > Log > Download.
- Select OMS from the Service and click OK.
- Set Host to the node for which the alarm is generated and the active OMS node.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information and files port_result.txt and ps_result.txt. Then, delete the two residual temporary files from the environment.
diff --git a/docs/mrs/umn/ALM-12053.html b/docs/mrs/umn/ALM-12053.html index 62f69a3a..f363ac07 100644 --- a/docs/mrs/umn/ALM-12053.html +++ b/docs/mrs/umn/ALM-12053.html @@ -66,11 +66,11 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Possible Causes
- The application process is abnormal. For example, the opened file or socket is not closed.
- The number of file handles cannot meet the current service requirements.
- The system is abnormal.
Procedure
Check information about files opened in processes.
-
- On FusionInsight Manager, click
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host for which the alarm is generated.
- Log in to the host for which the alarm is generated as user root.
- Run the lsof -n|awk '{print $2}'|sort|uniq -c|sort -nr|more command to check the process that occupies excessive file handles.
- Check whether the processes in which a large number of files are opened are normal. For example, check whether there are files or sockets not closed. +
- On FusionInsight Manager, click
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host for which the alarm is generated.
- Log in to the host for which the alarm is generated as user root.
- Run the lsof -n|awk '{print $2}'|sort|uniq -c|sort -nr|more command to check the process that occupies excessive file handles.
- Check whether the processes in which a large number of files are opened are normal. For example, check whether there are files or sockets not closed.
- Release the abnormal processes that occupy too many file handles.
- Five minutes later, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Increase the number of file handles.
-
- On FusionInsight Manager, click
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host for which the alarm is generated.
- Log in to the host for which the alarm is generated as user root.
- Contact the system administrator to increase the number of system file handles.
- Run the cat /proc/sys/fs/file-nr command to view the used handles and the maximum number of file handles. The first value is the number of used handles, the third value is the maximum number. Please check whether the usage exceeds the threshold.
- If yes, go to 9.
- If no, go to 11.
# cat /proc/sys/fs/file-nr +
- On FusionInsight Manager, click
in the row where the alarm is located in the real-time alarm list and obtain the IP address of the host for which the alarm is generated.
- Log in to the host for which the alarm is generated as user root.
- Contact the system administrator to increase the number of system file handles.
- Run the cat /proc/sys/fs/file-nr command to view the used handles and the maximum number of file handles. The first value is the number of used handles, the third value is the maximum number. Please check whether the usage exceeds the threshold.
- Wait for 5 minutes, and check whether the alarm is cleared.
@@ -80,7 +80,7 @@
- If yes, no further action is required.
- If no, go to 12.
- Wait for 5 minutes, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 14.
Collect fault information.
-+
- On the FusionInsight Manager home page of the active cluster, choose O&M > Log > Download.
- Select OMS from the Service and click OK.
- Set Host to the node for which the alarm is generated and the active OMS node.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On the FusionInsight Manager home page of the active cluster, choose O&M > Log > Download.
- Select OMS from the Service and click OK.
- Set Host to the node for which the alarm is generated and the active OMS node.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12054.html b/docs/mrs/umn/ALM-12054.html index 33a663c4..bf7cb2c3 100644 --- a/docs/mrs/umn/ALM-12054.html +++ b/docs/mrs/umn/ALM-12054.html @@ -2,7 +2,10 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-12054 Invalid Certificate File
Description
The system checks whether the certificate file is invalid (has expired or is not valid yet) on 23:00 every day. This alarm is generated when the certificate file is invalid.
-This alarm is cleared when a valid certificate is imported.
+This alarm is cleared when a valid certificate is imported and the alarm detection mechanism is triggered on the next hour.
+For MRS 3.2.0 or later, the certificate file is checked at the beginning of each hour.
+For versions earlier than MRS 3.2.0, the certificate file is checked on 23:00 every day.
+-Attribute
Alarm ID
@@ -65,8 +68,8 @@Possible Causes
No certificate (CA certificate, HA root certificate, HA user certificate, Gaussdb root certificate, or Gaussdb user certificate) is imported to the system, the certificate fails to be imported, or the certificate file is invalid.
Procedure
Check the alarm cause.
-
- On FusionInsight Manager, locate the target alarm in the real-time alarm list and click
.
View Additional Information to obtain the additional information about the alarm.
-+
- If CA Certificate is displayed in the additional alarm information, log in to the active OMS management node as user omm and go to 2.
- If HA root Certificate is displayed in the additional information, view Location to obtain the name of the host involved in this alarm. Then, log in to the host as user omm and go to 3.
- If HA server Certificate is displayed in the additional information, view Location to obtain the name of the host involved in this alarm. Then, log in to the host as user omm and go to 4.
- If Certificate has expired is displayed in the additional information, view Location to obtain the host name of the node for which the alarm is generated. Then, log in to the host as user omm and perform 2 to 4 to check whether the certificates have expired. If these certificates have not expired, check whether other certificates have been imported. If yes, import the certificate files again.
- On FusionInsight Manager, locate the target alarm in the real-time alarm list and click
.
View Additional Information to obtain the additional information about the alarm.
+
- If CA Certificate is displayed in the additional alarm information, log in to the active OMS management node as user omm and go to 2.
- If HA root Certificate is displayed in the additional information, view Location to obtain the name of the host involved in this alarm. Then, log in to the host as user omm and go to 3.
- If HA server Certificate is displayed in the additional information, view Location to obtain the name of the host involved in this alarm. Then, log in to the host as user omm and go to 4.
- If Certificate has expired is displayed in the additional information, view Location to obtain the name of the host for which the alarm is generated. Then, log in to the host as user omm and perform 2 to 4 in sequence to check whether the certificates have expired. If these certificates have not expired, check whether other certificates have been imported. If yes, import the certificate files again.
Check the validity period of the certificate files in the system.
- Check whether the current system time is in the validity period of the CA certificate.
Run the bash ${CONTROLLER_HOME}/security/cert/conf/querycertvalidity.sh command to check the effective time and due time of the CA root certificate.
@@ -94,7 +97,7 @@
- If yes, go to 7.
- If no, no further action is required.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller, OmmServer, OmmCore, and Tomcat, and click OK.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller, OmmServer, OmmCore, and Tomcat, and click OK.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-12055.html b/docs/mrs/umn/ALM-12055.html index 8862d12f..ec9f15bf 100644 --- a/docs/mrs/umn/ALM-12055.html +++ b/docs/mrs/umn/ALM-12055.html @@ -2,7 +2,10 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-12055 The Certificate File Is About to Expire
Description
The system checks the certificate file on 23:00 every day. This alarm is generated if the certificate file is about to expire within 30 days.
-This alarm is cleared when a certificate that is not about to expire is imported.
+This alarm is cleared when a certificate that is not about to expire is imported and the alarm detection mechanism is triggered on the next hour.
+For MRS 3.2.0 or later, the certificate file is checked at the beginning of each hour.
+For versions earlier than MRS 3.2.0, the certificate file is checked on 23:00 every day.
+-Attribute
Alarm ID
@@ -65,7 +68,7 @@Possible Causes
The remaining validity period of a system certificate (CA certificate, HA root certificate, HA user certificate, Gaussdb root certificate, or Gaussdb user certificate) is less than 30 days.
Procedure
Check the alarm cause.
-
- On FusionInsight Manager, locate the target alarm in the real-time alarm list and click
View Additional Information to obtain the additional information about the alarm.
+
- On FusionInsight Manager, locate the target alarm in the real-time alarm list and click
View Additional Information to obtain the additional information about the alarm.
- If CA Certificate is displayed in the additional alarm information, log in to the active OMS management node as user omm and go to 2.
- If HA root Certificate is displayed in the additional information, view Location to obtain the name of the host involved in this alarm. Then, log in to the host as user omm and go to 3.
- If HA server Certificate is displayed in the additional information, view Location to obtain the name of the host involved in this alarm. Then, log in to the host as user omm and go to 4.
Check the validity period of the certificate files in the system.
@@ -94,7 +97,7 @@
- If yes, go to 7.
- If no, no further action is required.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller, OmmServer, OmmCore, and Tomcat, and click OK.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller, OmmServer, OmmCore, and Tomcat, and click OK.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-12057.html b/docs/mrs/umn/ALM-12057.html index dfe37ff0..2af662f5 100644 --- a/docs/mrs/umn/ALM-12057.html +++ b/docs/mrs/umn/ALM-12057.html @@ -59,10 +59,10 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
-Possible Causes
Metadata is not configured with the task to periodically back up data to a third-party server.
Procedure
- On the FusionInsight Manager portal choose O&M > Alarm > Alarms.
- In the alarm list, click
in the row where the alarm is located and identify the data module from which the alarm is generated based on Additional Information.
- Choose O&M > Backup and Restoration > Backup Management > Create.
- Configure a backup task. The backup data to be configured is consistent with the data in Additional Information of the alarm.
- After the backup task is created successfully, wait for two minutes and check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 6.
Procedure
- On the FusionInsight Manager portal choose O&M > Alarm > Alarms.
- In the alarm list, click
in the row where the alarm is located and identify the data module from which the alarm is generated based on Additional Information.
- Choose O&M > Backup and Restoration > Backup Management > Create.
- Configure a backup task. The backup data to be configured is consistent with the data in Additional Information of the alarm.
- After the backup task is created successfully, wait for two minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- In the Service area, select Controller and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- In the Service area, select Controller and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12061.html b/docs/mrs/umn/ALM-12061.html index 64e0836a..0d68beb6 100644 --- a/docs/mrs/umn/ALM-12061.html +++ b/docs/mrs/umn/ALM-12061.html @@ -69,7 +69,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
The alarm is generated when the process usage exceeds the threshold for the times specified by Trigger Count.
Set the alarm threshold based on the actual process usage. To check the process usage, choose O&M > Alarm > Thresholds > Name of the desired cluster > Host> Process > omm Process Usage, as shown in Figure 1.
- +- 2 minutes later, check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 3.
Check whether the maximum number of processes (including threads) opened by user omm is appropriate.
@@ -80,7 +80,7 @@
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and view the IP address of the host for which the alarm is generated.
- Log in to the host where the alarm is generated as user root.
- Run the ps -o nlwp, pid, lwp, args, -u omm|sort -n command to check the numbers of threads used by the system. The result is sorted based on the thread number. Analyze the top 5 thread numbers and check whether the threads are incorrectly used. If they are, contact maintenance personnel to rectify the fault. If they are not, run the ulimit -u command to change the maximum number to be greater than 60000.
- Five minutes later, check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 12.
Collect fault information.
-+
- On the FusionInsight Manager home page of the active clusters, choose O&M > Log > Download.
- Select OmmServer and NodeAgent from the Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On the FusionInsight Manager home page of the active clusters, choose O&M > Log > Download.
- Select OmmServer and NodeAgent from the Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12062.html b/docs/mrs/umn/ALM-12062.html index aea4acd9..b93ec017 100644 --- a/docs/mrs/umn/ALM-12062.html +++ b/docs/mrs/umn/ALM-12062.html @@ -62,7 +62,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and view the IP address of the host for which the alarm is generated.
- Log in to the host where the alarm is generated as user root.
- Run the su - omm command to switch to user omm.
- Run the vi $BIGDATA_LOG_HOME/controller/scriptlog/modify_manager_param.log command to open the log file and search for the log file containing the following information: Current oms configurations cannot support xx nodes. In the information, xx indicates the number of nodes in the cluster.
- Optimize the current cluster configuration by following the instructions in Optimizing Manager Configurations Based on the Number of Cluster Nodes.
- One hour later, check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 7.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller from the Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller from the Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12063.html b/docs/mrs/umn/ALM-12063.html index 323d6780..f5845fde 100644 --- a/docs/mrs/umn/ALM-12063.html +++ b/docs/mrs/umn/ALM-12063.html @@ -71,7 +71,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Contact hardware engineers to rectify the disk.
- One hour later, check whether this alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 8.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent from the Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent from the Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12064.html b/docs/mrs/umn/ALM-12064.html index 2401a77c..ab97f4d0 100644 --- a/docs/mrs/umn/ALM-12064.html +++ b/docs/mrs/umn/ALM-12064.html @@ -63,7 +63,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Run the vim /etc/sysctl.conf command to change the value of net.ipv4.ip_local_port_range to 32768 61000. If this parameter does not exist, add the following configuration: net.ipv4.ip_local_port_range = 32768 61000.
- Run the sysctl -p /etc/sysctl.conf command for the modification to take effect.
- One hour later, check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 7.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12066.html b/docs/mrs/umn/ALM-12066.html index 52a6073c..5c660d68 100644 --- a/docs/mrs/umn/ALM-12066.html +++ b/docs/mrs/umn/ALM-12066.html @@ -59,7 +59,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Possible Causes
- The /etc/ssh/sshd_config configuration file is damaged.
- The password of user omm has expired.
Procedure
Check the status of the /etc/ssh/sshd_config configuration file.
-
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm and click
to view the host list in the alarm details.
- Log in to the active OMS node as user omm.
- Run the ssh command, for example, ssh host2, on each node in the alarm details to check whether the connection fails. (host2 is a node other than the OMS node in the alarm details.) +
@@ -69,17 +69,17 @@
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm and click
to view the host list in the alarm details.
- Log in to the active OMS node as user omm.
- Run the ssh command, for example, ssh host2, on each node in the alarm details to check whether the connection fails. (host2 is a node other than the OMS node in the alarm details.)
- Open the /etc/ssh/sshd_config configuration file on host2 and check whether AllowUsers or DenyUsers is configured for other nodes.
- If yes, go to 5.
- If no, contact OS experts.
- Modify the whitelist or blacklist to ensure that user omm is in the whitelist or not in the blacklist. Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
- Add the public key of user omm of the peer host to the trust list of the local host. Run the ssh command, for example, ssh host2, on each node in the alarm details to check whether the connection fails. (host2 is a node other than the OMS node in the alarm details.)
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select Controller for Service and click OK.
- Click
in the upper right corner to set the log collection time range. Generally, the time range is 10 minutes before and after the alarm generation time. Click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select Controller for Service and click OK.
- Click
in the upper right corner to set the log collection time range. Generally, the time range is 10 minutes before and after the alarm generation time. Click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
diff --git a/docs/mrs/umn/ALM-12067.html b/docs/mrs/umn/ALM-12067.html index 1297f809..41e1b008 100644 --- a/docs/mrs/umn/ALM-12067.html +++ b/docs/mrs/umn/ALM-12067.html @@ -61,10 +61,10 @@Related Information
Perform the following steps to handle abnormal trust relationships between nodes:
-
- Perform this operation as user omm.
- If the network between nodes is disconnected, rectify the network fault first. Check whether the two nodes are connected to the same security group and whether hosts.deny and hosts.allow are set.
- Run the ssh-add -l command on both nodes to check whether any identities exist.
+
- Run the ssh-add -l command on both nodes to check whether any identities exist.
-
- If no identities are displayed, run the ps -ef|grep ssh-agent command to find the ssh-agent process, stop the process, and wait for the process to automatically restart.
-
- Run the ssh-add -l command to check whether the identities have been added. If yes, manually run the ssh command to check whether the trust relationship is normal.
+
- If no identities are displayed, run the ps -ef|grep ssh-agent command to find the ssh-agent process, stop the process, and wait for the process to automatically restart.
+
- Run the ssh-add -l command to check whether the identities have been added. If yes, manually run the ssh command to check whether the trust relationship is normal.
- If identities exist, check whether the /home/omm/.ssh/authorized_keys file contains the information in the /home/omm/.ssh/id_rsa.pub file of the peer node. If it does not, manually add the information.
- Check whether the permissions on the files in the /home/omm/.ssh directory are modified.
- Check the /var/log/Bigdata/nodeagent/scriptlog/ssh-agent-monitor.log file.
- If the /home directory of user omm is deleted, contact MRS support personnel for assistance.
Possible Causes
- The Tomcat directory permission is abnormal, and the Tomcat process is abnormal.
Procedure
Check whether the permission on the Tomcat directory is normal.
-
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and click
to view the IP address of the host for which the alarm is generated.
- Log in to the alarm host as user root.
- Run the su - omm command to switch to user omm.
- Run the vi $BIGDATA_LOG_HOME/omm/oms/ha/scriptlog/tomcat.log command to check whether the Tomcat resource log contains keyword Cannot find XXX and rectify the file permission based on the keyword.
- After 5 minutes, check whether the alarm is automatically cleared.
+
- If yes, no further action is required.
- If no, go to 6.
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and click
to view the IP address of the host for which the alarm is generated.
- Log in to the alarm host as user root.
- Run the su - omm command to switch to user omm.
- Run the vi $BIGDATA_LOG_HOME/omm/oms/ha/scriptlog/tomcat.log command to check whether the Tomcat resource log contains keyword Cannot find XXX and rectify the file permission based on the keyword.
- After 5 minutes, check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select OmmServer and Tomcat, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select OmmServer and Tomcat, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-12068.html b/docs/mrs/umn/ALM-12068.html index e6aa575d..cef06028 100644 --- a/docs/mrs/umn/ALM-12068.html +++ b/docs/mrs/umn/ALM-12068.html @@ -61,11 +61,11 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
Possible Causes
The ACS process is abnormal.
Procedure
Check whether the ACS process is normal.
-
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and click
to view the name of the host for which the alarm is generated.
- Log in to the alarm host as user root.
- Run the su - omm command and then sh ${BIGDATA_HOME}/om-server/OMS/workspace0/ha/module/hacom/script/status_ha.sh to check whether the status of the ACS resources managed by the HA is normal. In the single-node system, the ACS resource is in the normal state. In the dual-node system, the ACS resource is in the normal state on the active node and in the stopped state on the standby node. +
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and click
to view the name of the host for which the alarm is generated.
- Log in to the alarm host as user root.
- Run the su - omm command and then sh ${BIGDATA_HOME}/om-server/OMS/workspace0/ha/module/hacom/script/status_ha.sh to check whether the status of the ACS resources managed by the HA is normal. In the single-node system, the ACS resource is in the normal state. In the dual-node system, the ACS resource is in the normal state on the active node and in the stopped state on the standby node.
- Run the vi $BIGDATA_LOG_HOME/omm/oms/ha/scriptlog/acs.log command to check whether the ACS resource log of HA contains the keyword ERROR. If yes, analyze the logs to locate the resource exception cause and fix the exception.
- After 5 minutes, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller and OmmServer, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller and OmmServer, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-12069.html b/docs/mrs/umn/ALM-12069.html index 83d58e67..cea673d1 100644 --- a/docs/mrs/umn/ALM-12069.html +++ b/docs/mrs/umn/ALM-12069.html @@ -61,11 +61,11 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
Possible Causes
The AOS process is abnormal.
Procedure
Check whether the AOS process is normal.
-
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and click
to view the name of the host for which the alarm is generated.
- Log in to the alarm host as user root.
- Run the sh ${BIGDATA_HOME}/om-server/OMS/workspace0/ha/module/hacom/script/status_ha.sh command to check whether the status of the AOS resources managed by the HA is normal. In the single-node system, the AOS resource is in the normal state. In the dual-node system, the AOS resource is in the normal state on the active node and in the stopped state on the standby node. +
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and click
to view the name of the host for which the alarm is generated.
- Log in to the alarm host as user root.
- Run the sh ${BIGDATA_HOME}/om-server/OMS/workspace0/ha/module/hacom/script/status_ha.sh command to check whether the status of the AOS resources managed by the HA is normal. In the single-node system, the AOS resource is in the normal state. In the dual-node system, the AOS resource is in the normal state on the active node and in the stopped state on the standby node.
- Run the vi $BIGDATA_LOG_HOME/omm/oms/ha/scriptlog/aos.log command to check whether the AOS resource log of HA contains the keyword ERROR. If yes, analyze the logs to locate the resource exception cause and fix the exception.
- After 5 minutes, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller and OmmServer, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller and OmmServer, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-12070.html b/docs/mrs/umn/ALM-12070.html index 9af498b1..c8a56a10 100644 --- a/docs/mrs/umn/ALM-12070.html +++ b/docs/mrs/umn/ALM-12070.html @@ -65,7 +65,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Run the vi $BIGDATA_LOG_HOME/omm/oms/ha/scriptlog/controller.log command to view the Controller resource logs, and run the vi $BIGDATA_LOG_HOME/controller/controller.log command to view the Controller running logs, check whether the keyword ERROR exists. Analyze the logs to locate and rectify the fault.
- Five minutes later, check whether this alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 6.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller and OmmServe for Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour before and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller and OmmServe for Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour before and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12071.html b/docs/mrs/umn/ALM-12071.html index edc7c749..3d96f959 100644 --- a/docs/mrs/umn/ALM-12071.html +++ b/docs/mrs/umn/ALM-12071.html @@ -65,7 +65,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Run the vi $BIGDATA_LOG_HOME/omm/oms/ha/scriptlog/httpd.log command to view the httpd resource logs, check whether the keyword ERROR exists. Analyze the logs to locate and rectify the fault.
- Five minutes later, check whether this alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 7.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12072.html b/docs/mrs/umn/ALM-12072.html index b630afdb..64ccb756 100644 --- a/docs/mrs/umn/ALM-12072.html +++ b/docs/mrs/umn/ALM-12072.html @@ -70,7 +70,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the ifconfig NIC name Floating IPaddress netmask Subnet mask command to reconfigure the NIC with the floating IP address. (For example, ifconfig eth0 10.10.10.102 netmask 255.255.255.0).
- Five minutes later, check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 8.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12073.html b/docs/mrs/umn/ALM-12073.html index 6c3fba8c..406d74f4 100644 --- a/docs/mrs/umn/ALM-12073.html +++ b/docs/mrs/umn/ALM-12073.html @@ -65,7 +65,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the vi $BIGDATA_LOG_HOME/omm/oms/cep/cep.log and vi $BIGDATA_LOG_HOME/omm/oms/cep/scriptlog/cep_ha.log commands to view the CEP resource logs, check whether the keyword ERROR exists. Analyze the logs to locate and rectify the fault.
- Five minutes later, check whether this alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 6.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12074.html b/docs/mrs/umn/ALM-12074.html index 885a1f9f..661d29ed 100644 --- a/docs/mrs/umn/ALM-12074.html +++ b/docs/mrs/umn/ALM-12074.html @@ -65,7 +65,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the vi $BIGDATA_LOG_HOME/omm/oms/fms/fms.log and vi $BIGDATA_LOG_HOME/omm/oms/fms/scriptlog/fms_ha.log commands to view the FMS resource logs, check whether the keyword ERROR exists. Analyze the logs to locate and rectify the fault.
- 5 minutes later, check whether this alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 6.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M> Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M> Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12075.html b/docs/mrs/umn/ALM-12075.html index be5b158a..3d916ff6 100644 --- a/docs/mrs/umn/ALM-12075.html +++ b/docs/mrs/umn/ALM-12075.html @@ -65,7 +65,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the vi $BIGDATA_LOG_HOME/omm/oms/pms/pms.log and vi $BIGDATA_LOG_HOME/omm/oms/pms/scriptlog/pms_ha.log commands to view the PMS resource logs, check whether the keyword ERROR exists. Analyze the logs to locate and rectify the fault.
- Five minutes later, check whether this alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 6.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M> Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M> Log > Download.
- Select Controller and OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 1 hour before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12076.html b/docs/mrs/umn/ALM-12076.html index 4d1b04b3..ad5bef2e 100644 --- a/docs/mrs/umn/ALM-12076.html +++ b/docs/mrs/umn/ALM-12076.html @@ -83,11 +83,11 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Contact the network administrator to check whether the network is faulty.
- Five minutes later, check whether the alarm is cleared.
-
- If it is, no further action is required.
- If it is not, go to 4.
- Log in to the active and standby management nodes, run the su -omm command to switch to user omm, go to the ${BIGDATA_HOME} /om-server/om/sbin/ directory, and run the status-oms.sh script to check whether the floating IP addresses and GaussDB resources of the active and standby FusionInsight Managers are in the status shown in the following figure.
+
- Log in to the active and standby management nodes, run the su -omm command to switch to user omm, go to the ${BIGDATA_HOME} /om-server/om/sbin/ directory, and run the status-oms.sh script to check whether the floating IP addresses and GaussDB resources of the active and standby FusionInsight Managers are in the status shown in the following figure.
- If they are, find the alarm in the alarm list and manually clear the alarm.
- If they are not, go to 5.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select OmmServer for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12077.html b/docs/mrs/umn/ALM-12077.html index 08e8a423..e5ecc3e7 100644 --- a/docs/mrs/umn/ALM-12077.html +++ b/docs/mrs/umn/ALM-12077.html @@ -67,7 +67,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the chage -E 'yyyy-MM-dd' omm command to set the expiration time of user omm. Eight hours later, check whether the alarm is automatically cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12078.html b/docs/mrs/umn/ALM-12078.html index 7335c861..ce4a479f 100644 --- a/docs/mrs/umn/ALM-12078.html +++ b/docs/mrs/umn/ALM-12078.html @@ -67,7 +67,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the chage -M 'days' omm command to set the validity period of the password for user omm. Eight hours later, check whether the alarm is automatically cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M> Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M> Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12079.html b/docs/mrs/umn/ALM-12079.html index eb13fe2f..fcbc2fa4 100644 --- a/docs/mrs/umn/ALM-12079.html +++ b/docs/mrs/umn/ALM-12079.html @@ -67,7 +67,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the chage -E 'yyyy-MM-dd' omm command to set the validity period of user omm. Eight hours later, check whether the alarm is automatically cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12080.html b/docs/mrs/umn/ALM-12080.html index 9bffa818..13cf3908 100644 --- a/docs/mrs/umn/ALM-12080.html +++ b/docs/mrs/umn/ALM-12080.html @@ -67,7 +67,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the chage -M 'days' omm command to set the validity period of the password for user omm. Eight hours later, check whether the alarm is automatically cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M> Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M> Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12081.html b/docs/mrs/umn/ALM-12081.html index ac696976..a76170aa 100644 --- a/docs/mrs/umn/ALM-12081.html +++ b/docs/mrs/umn/ALM-12081.html @@ -67,7 +67,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the chage -E 'yyyy-MM-dd' omm command to set the validity period of user ommdba. Eight hours later, check whether the alarm is automatically cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12082.html b/docs/mrs/umn/ALM-12082.html index 6655db75..a02f0a50 100644 --- a/docs/mrs/umn/ALM-12082.html +++ b/docs/mrs/umn/ALM-12082.html @@ -67,7 +67,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the chage -E 'yyyy-MM-dd' ommdba command to set the validity period of user ommdba. Eight hours later, check whether the alarm is automatically cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12083.html b/docs/mrs/umn/ALM-12083.html index 906f1711..593f5edf 100644 --- a/docs/mrs/umn/ALM-12083.html +++ b/docs/mrs/umn/ALM-12083.html @@ -67,7 +67,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the chage -M 'days' ommdba command to set the validity period of the password for user ommdba. Eight hours later, check whether the alarm is automatically cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12084.html b/docs/mrs/umn/ALM-12084.html index 4f5ffe24..a66079f1 100644 --- a/docs/mrs/umn/ALM-12084.html +++ b/docs/mrs/umn/ALM-12084.html @@ -67,7 +67,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Run the chage -M 'days' ommdba command to set the validity period of the password for user ommdba. Eight hours later, check whether the alarm is automatically cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select NodeAgent for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12085.html b/docs/mrs/umn/ALM-12085.html index 710e43f2..22b1bf20 100644 --- a/docs/mrs/umn/ALM-12085.html +++ b/docs/mrs/umn/ALM-12085.html @@ -75,7 +75,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Expand the capacity of the node
- In the next execution period, 03:00, check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 15.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M> Log > Download.
- Select Controller for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On FusionInsight Manager, choose O&M> Log > Download.
- Select Controller for Service and click OK.
- Click
in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12087.html b/docs/mrs/umn/ALM-12087.html index 6a998110..ffe0d119 100644 --- a/docs/mrs/umn/ALM-12087.html +++ b/docs/mrs/umn/ALM-12087.html @@ -66,12 +66,12 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
Procedure
Check whether the system is in the upgrade observation period.
- Log in to the active management node as user root.
- Run the following commands to switch to user omm and log in to the omm database:
su - omm
gsql -U omm -W omm database password -p 20015
-- Run the select * from OM_CLUSTERS command to view cluster information.
- Check whether the value of upgradObservationPeriod isON is true, as shown in Figure 1.
- If it is, the system is in the upgrade observation period. Use the UpdateTool to submit the upgrade task. For details, see the upgrade guide of the corresponding version.
- If it is not, go to 6. +
- Run the select * from OM_CLUSTERS command to view cluster information.
- Check whether the value of upgradObservationPeriod isON is true, as shown in Figure 1.
- If it is, the system is in the upgrade observation period. Use the UpdateTool to submit the upgrade task. For details, see the upgrade guide of the corresponding version.
- If it is not, go to 6.
- In the early morning of the next day, check whether this alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 6.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Controller from the Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Controller from the Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12089.html b/docs/mrs/umn/ALM-12089.html index 2693e7db..9962d25b 100644 --- a/docs/mrs/umn/ALM-12089.html +++ b/docs/mrs/umn/ALM-12089.html @@ -67,7 +67,7 @@Alarm Clearing
This alarm will be automatically cleared after the fault is rectified.
- Remove the file $NODE_AGENT_HOME/etc/agent/hosts.ini of all nodes in the cluster, and clean up the file /var/log/Bigdata/unreachable/unreachable_ip_info.log, and then manually clear the alarm.
- Wait for 30 seconds and checking if the alarm was been cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select OmmAgent from the Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select OmmAgent from the Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-12101.html b/docs/mrs/umn/ALM-12101.html index 5031e4fd..c4577f74 100644 --- a/docs/mrs/umn/ALM-12101.html +++ b/docs/mrs/umn/ALM-12101.html @@ -14,7 +14,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- 12101
Major
+Critical
diff --git a/docs/mrs/umn/ALM-12102.html b/docs/mrs/umn/ALM-12102.html index 96f584f2..219cf47f 100644 --- a/docs/mrs/umn/ALM-12102.html +++ b/docs/mrs/umn/ALM-12102.html @@ -49,7 +49,7 @@ Yes
Possible Causes
The roles of the components that support DR are not deployed based on DR requirements.
Procedure
Obtain alarm information.
-+
- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, click
in the row that contains the alarm and view the roles that are not deployed based on DR requirements in Additional Information.
- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, click
in the row that contains the alarm and view the roles that are not deployed based on DR requirements in Additional Information.
Redeploy the role instance.
diff --git a/docs/mrs/umn/ALM-12103.html b/docs/mrs/umn/ALM-12103.html new file mode 100644 index 00000000..8cf6e245 --- /dev/null +++ b/docs/mrs/umn/ALM-12103.html @@ -0,0 +1,77 @@ + + +
- Choose Cluster > Services > Name of the desired service > Instance. On the instance page, redeploy or adjust the role instance.
- Check whether the alarm is cleared 10 minutes later.
- If yes, no further action is required.
- If no, contact O&M personnel.
ALM-12103 Executor Resource Exception
+++Description
HA checks the Executor resources of Manager every 30 seconds. This alarm is generated when HA detects that the Executor resources are abnormal for two consecutive times.
+This alarm is cleared when the Executor resources are normal.
+Resource Type of Executor is Single-active. Active/standby will be triggered upon resource exceptions. When this alarm is generated, the active/standby switchover is complete and new Executor resources have been enabled on the new active Manager. In this case, this alarm is cleared. This alarm is used to notify users of the cause of the active/standby Manager switchover.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Auto Clear
++ + + 12103
++ Major
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster or system for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + + HostName
++ Specifies the host for which the alarm is generated.
++Impact on the System
+
- The active/standby Manager switchover occurs.
- The Executor process keeps restarting. As a result, the cluster page may fail to be accessed.
+Possible Causes
The Executor process is abnormal.
++ +Procedure
Check whether the Executor process is abnormal.
++
- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and click
to view the name of the host for which the alarm is generated.
- Log in to the host for which the alarm is generated as user root.
- Run the su - omm command to switch to user omm.
- Run the sh ${BIGDATA_HOME}/om-server/OMS/workspace0/ha/module/hacom/script/status_ha.sh command to check whether the status of the Executor resources managed by the HA is normal. In the single-node system, the Executor resource is in the normal state. In the dual-node system, the Executor resource is in the normal state on the active node and in the stopped state on the standby node. +
- Run the vi $BIGDATA_LOG_HOME/omm/oms/ha/scriptlog/executor.log command to check whether the Executor resource log of HA contains the keyword ERROR. If yes, analyze the log to locate the resource exception cause and fix the exception.
- After 5 minutes, check whether this alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 7.
Collect the fault information.
++
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller and OmmServer, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
++ diff --git a/docs/mrs/umn/ALM-12104.html b/docs/mrs/umn/ALM-12104.html new file mode 100644 index 00000000..5fcf3cde --- /dev/null +++ b/docs/mrs/umn/ALM-12104.html @@ -0,0 +1,76 @@ + + +++Parent topic: Alarm Reference (Applicable to MRS 3.x)+ALM-12104 Abnormal Knox Resources
+++Description
HA checks the Knox resources of Manager every 70 seconds. This alarm is generated when HA detects that the Knox resources are abnormal for three consecutive times.
+This alarm is cleared when HA detects that the Knox resources are normal.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Auto Clear
++ + + 12104
++ Major
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster or system for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + + HostName
++ Specifies the host for which the alarm is generated.
++Impact on the System
Requests sent by upper-layer services by using Knox cannot be properly processed.
++Possible Causes
The Knox process is abnormal.
++ +Procedure
Check whether the Knox process is normal.
++
- Log in to FusionInsight Manager. In the alarm list, locate the row that contains the alarm and view the name of the host for which the alarm is generated.
- Use PuTTY to log in to the host for which the alarm is generated as user root.
- Run the su - omm command to switch to user omm.
- Run the sh ${BIGDATA_HOME}/om-server/om/sbin/status-oms.sh command to check whether the status of the Knox resources managed by HA is normal. If the status is normal, the Knox resources are normal. Otherwise, the Knox resources are abnormal. +
- Run the vi $BIGDATA_LOG_HOME/omm/oms/ha/scriptlog/knox.log command to check whether the Knox resource log of HA contains the keyword ERROR. If yes, analyze the log to locate the resource exception cause and fix the exception.
- After 5 minutes, check whether this alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 7.
Collect the fault information.
++
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Services area, select Controller and OmmServer, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
++ diff --git a/docs/mrs/umn/ALM-12110.html b/docs/mrs/umn/ALM-12110.html index 77c69a1d..7e15d91f 100644 --- a/docs/mrs/umn/ALM-12110.html +++ b/docs/mrs/umn/ALM-12110.html @@ -59,7 +59,7 @@++Parent topic: Alarm Reference (Applicable to MRS 3.x)+Possible Causes
- The meta role of the MRS cluster is abnormal.
- The cluster has been bound to an agency and accessed OBS but has been unbound from the agency. As a result, the cluster has not been bound to any agency.
Procedure
Check the status of the meta role.
-
- On FusionInsight Manager of the cluster, choose O&M > Alarm > Alarms. On the page that is displayed, click
in the row containing the alarm, and determine the IP address of the host for which the alarm is generated.
- On FusionInsight Manager of the cluster, choose Cluster > Services > Meta. On the page that is displayed, click the Instance tab, and check whether the meta role corresponding to the host for which the alarm is generated is normal. +
- On FusionInsight Manager of the cluster, choose O&M > Alarm > Alarms. On the page that is displayed, click
in the row containing the alarm, and determine the IP address of the host for which the alarm is generated.
- On FusionInsight Manager of the cluster, choose Cluster > Services > Meta. On the page that is displayed, click the Instance tab, and check whether the meta role corresponding to the host for which the alarm is generated is normal.
- Select the abnormal role and choose More > Restart Instance to restart the abnormal meta role. After the restart is complete, check whether the alarm is cleared several minutes later.
- If yes, no further action is required.
- If no, go to 4.
Rebind the cluster to an agency.
diff --git a/docs/mrs/umn/ALM-12172.html b/docs/mrs/umn/ALM-12172.html new file mode 100644 index 00000000..d4097225 --- /dev/null +++ b/docs/mrs/umn/ALM-12172.html @@ -0,0 +1,74 @@ + + +ALM-12172 Failed to Report Metrics to Cloud Eye
+++Description
After metric sharing is enabled for a cluster, the Controller periodically collects cluster metrics and reports them to Cloud Eye.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Auto Clear
++ + + 12172
++ Major
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster or system for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + + HostName
++ Specifies the host for which the alarm is generated.
++Impact on the System
MRS monitoring metrics are unavailable on Cloud Eye.
++Possible Causes
+
- Failed to call Cloud Eye APIs due to insufficient permissions.
- Failed to report data to Cloud Eye due to network problems.
- Failed to report data to Cloud Eye due to internal errors.
+Procedure
+
- On FusionInsight Manager, choose O&M > Alarm > Alarms. On the page that is displayed, click
in the row containing the alarm and view the additional information of the alarm.
- Rectify the fault based on the following scenarios:
+
- If "Call CES to send metrics fail. Permission exception" is displayed in the additional information, the token of the resource tenant is invalid. Restart the Controller and obtain the token again.
- If "Call CES to send metrics fail. Request CES error code xxx" is displayed in the additional information, an error occurs in the request to Cloud Eye. Check the network connectivity and authentication information.
- If "Call CES to send metrics fail. CES internal error code xxx" is displayed in the additional information, the Cloud Eye service encounters an internal error and is unavailable. Contact O&M personnel and send collected fault logs.
+Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
++Related Information
None
+++ diff --git a/docs/mrs/umn/ALM-12180.html b/docs/mrs/umn/ALM-12180.html index e9904d43..b7725049 100644 --- a/docs/mrs/umn/ALM-12180.html +++ b/docs/mrs/umn/ALM-12180.html @@ -2,23 +2,28 @@++Parent topic: Alarm Reference (Applicable to MRS 3.x)+ALM-12180 Suspended Disk I/O
@@ -86,7 +91,7 @@Description
- For HDDs, the alarm is triggered when any of the following conditions is met:
-
- The system collects data every 3 seconds, and detects that the svctm value exceeds 6s for 10 consecutive periods within 30 seconds.
- The system collects data every 3 seconds, and detects that the avgqu-sz value is greater than 0, the IOPS or bandwidth is 0, and the ioutil value is greater than 99% for 10 consecutive periods within 30 seconds.
- For SSDs, the alarm is triggered when any of the following conditions is met:
+
- The system collects data every 3 seconds, and detects that the svctm value exceeds 3s for 10 consecutive periods within 30 seconds.
- The system collects data every 3 seconds, and detects that the avgqu-sz value is greater than 0, the IOPS or bandwidth is 0, and the ioutil value is greater than 99% for 10 consecutive periods within 30 seconds.
- For SSDs, the alarm is triggered when any of the following conditions is met:
- The system collects data every 3 seconds, and detects that the svctm value exceeds 2s for 10 consecutive periods within 30 seconds.
- The system collects data every 3 seconds, and detects that the avgqu-sz value is greater than 0, the IOPS or bandwidth is 0, and the ioutil value is greater than 99% for 10 consecutive periods within 30 seconds.
This alarm is automatically cleared when the preceding conditions have not been met for 90s.
- Run the following command in the OS to collect data: -
+
Parameters are as follows:
--
- avgqu-sz indicates the disk queue depth.
- The sum of r/s and w/s is the IOPS.
- The sum of rkB/s and wkB/s is the bandwidth.
- %util is the ioutil value.
- The formula for calculating svctm is as follows:
svctm = (tot_ticks_new - tot_ticks_old) / (rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old)
+avgqu-sz indicates the disk queue depth.
+The sum of r/s and w/s is the IOPS.
+The sum of rkB/s and wkB/s is the bandwidth.
+%util is the ioutil value.
+- MRS 3.1.0:
Run the iostat -x -t command in the OS.
++
- Calculate svctm as follows in versions later than MRS 3.1.0:
svctm = (tot_ticks_new - tot_ticks_old)/(rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old)
If rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old is 0, then svctm is 0.
The parameters can be obtained as follows:
The system runs the cat /proc/diskstats command every 3 seconds to collect data. For example:
-+
In these two commands:
In the data collected for the first time, the number in the fourth column is the rd_ios_old value, the number in the eighth column is the wr_ios_old value, and the number in the thirteenth column is the tot_ticks_old value.
In the data collected for the second time, the number in the fourth column is the rd_ios_new value, the number in the eighth column is the wr_ios_new value, and the number in the thirteenth column is the tot_ticks_new value.
In this case, the value of svctm is as follows:
-(19571460 - 19569526) / (1101553 + 28747977 - 1101553 - 28744856) = 0.6197
+(19571460 - 19569526)/(1101553 + 28747977 - 1101553 - 28744856) = 0.6197
- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms.
- View the detailed information about the alarm. Check the values of HostName and DiskName in the location information to obtain the information about the faulty disk for which the alarm is reported.
- Replace the hard disk.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select OMS for Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select OMS for Service and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-12190.html b/docs/mrs/umn/ALM-12190.html new file mode 100644 index 00000000..79b7a2c2 --- /dev/null +++ b/docs/mrs/umn/ALM-12190.html @@ -0,0 +1,87 @@ + + +Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-12190 Number of Knox Connections Exceeds the Threshold
+++Description
The system periodically checks the number of connections to all Knox topologies. This alarm is generated when the number of connections to a topology exceeds the threshold (90% by default). This alarm is automatically cleared when the number of connections to a topology falls below the threshold.
++This alarm applies to clusters of MRS 3.1.0 or later.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Auto Clear
++ + + 12190
++ Major
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster or system for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + HostName
++ Specifies the host for which the alarm is generated.
++ + + Topology
++ Specifies the Knox topology for which the alarm is generated.
++Impact on the System
The topology may reach the upper limit of connections and fail to forward requests, adversely affecting the MRS functions.
++Possible Causes
Hue or Manager is too frequently used, but the default maximum number of Knox connections is small.
++Procedure
+
- Log in to active and standby OMS nodes as user root, respectively.
- Add the following configuration to the gateway-site.xml file on the active and standby OMS nodes to increase the number of thread pools:
vi /opt/knox/conf/gateway-site.xml
+<property> +<name>gateway.httpclient.maxConnections</name> +<value>64</value> +</property>+- Log in to the active OMS node as user omm and run the following command to restart the Knox process:
sh /opt/knox/bin/restart-knox.sh
+- After 5 minutes, check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 5.
- Contact O&M personnel to rectify the fault.
+Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
++Related Information
None
+++ diff --git a/docs/mrs/umn/ALM-13000.html b/docs/mrs/umn/ALM-13000.html index b5bd5013..2710fd33 100644 --- a/docs/mrs/umn/ALM-13000.html +++ b/docs/mrs/umn/ALM-13000.html @@ -85,7 +85,7 @@++Parent topic: Alarm Reference (Applicable to MRS 3.x)+Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service: (KrbServer logs do not need to be downloaded in normal mode.)
-
- ZooKeeper
- KrbServer
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-13001.html b/docs/mrs/umn/ALM-13001.html index 2fdd21a8..54c9f2ea 100644 --- a/docs/mrs/umn/ALM-13001.html +++ b/docs/mrs/umn/ALM-13001.html @@ -74,7 +74,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > quorumpeer > Performance and increase the value of maxCnxns as required.
- Save the configuration and restart the ZooKeeper service.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 15.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service:
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service:
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected log information.
diff --git a/docs/mrs/umn/ALM-13002.html b/docs/mrs/umn/ALM-13002.html index b1e6d7b1..7db4f4ee 100644 --- a/docs/mrs/umn/ALM-13002.html +++ b/docs/mrs/umn/ALM-13002.html @@ -71,7 +71,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-13003.html b/docs/mrs/umn/ALM-13003.html index 3cfd8d35..ab82ece8 100644 --- a/docs/mrs/umn/ALM-13003.html +++ b/docs/mrs/umn/ALM-13003.html @@ -72,7 +72,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Save the configuration and restart the ZooKeeper service.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ZooKeeper for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ZooKeeper for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-13004.html b/docs/mrs/umn/ALM-13004.html index 838b11e0..de5c4887 100644 --- a/docs/mrs/umn/ALM-13004.html +++ b/docs/mrs/umn/ALM-13004.html @@ -70,7 +70,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Save the configuration and restart the ZooKeeper service.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-13005.html b/docs/mrs/umn/ALM-13005.html index 1074b33d..6df02061 100644 --- a/docs/mrs/umn/ALM-13005.html +++ b/docs/mrs/umn/ALM-13005.html @@ -105,7 +105,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- View the component of the alarm directory in the table, open the corresponding service page, and choose Configurations > All Configurations. On the displayed page, search for zk.quota in the upper right corner. The search result is the quota of the alarm directory.
- Check whether the quota of the alarm directory for which the alarm is reported is appropriate. The quota must be greater than or equal to the actual value, which can be obtained in Trigger Condition.
- Modify the services.quota value as prompted and save the configuration.
- After the time specified by service.quotas.auto.check.cron.expression, check whether the alarm is cleared.
- If it is, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-13006.html b/docs/mrs/umn/ALM-13006.html index ff7ed2df..fc4a891f 100644 --- a/docs/mrs/umn/ALM-13006.html +++ b/docs/mrs/umn/ALM-13006.html @@ -60,10 +60,10 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Procedure
Check whether a large amount of data is written to the directory for which the alarm is generated.
- On FusionInsight Manager, choose O&M > Alarm > Alarms. On the displayed interface, click the drop-down button of Znode Number or Capacity Exceeds the Threshold. Confirm the Znode for which the alarm is generated in Location Information.
- Log in to FusionInsight Manager, open the ZooKeeper service interface, and select Resource. In the table Used Resources (By Second-Level Znode), check whether a large amount of data is written to the top-level Znode for which the alarm is reported. -
- Log in to the ZooKeeper client and delete the data in the top-level Znode.
- Log in to FusionInsight Manager and open the ZooKeeper service interface. On the Resource page, choose
> By Znode quantity in Used Resources (By Second-Level Znode). Threshold Configuration of By Znode quantity is displayed. Click Modify under Operation. Increase the threshold by referringto the value of max.Znode.count by choosing Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > Quota.
- In the Used Resources (By Second-Level Znode), choose
> By capacity. The Threshold Settings page of By Capacity is displayed. Click Modify under Operation. Increase the threshold by referring to the value of max.data.size by choosing Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > Quota.
- Check whether the alarm is cleared.
+
- If it is, no further action is required.
- If it is not, go to 7.
- Log in to the ZooKeeper client and delete the data in the top-level Znode.
- Log in to FusionInsight Manager and open the ZooKeeper service interface. On the Resource page, choose
> By Znode quantity in Used Resources (By Second-Level Znode). Threshold Configuration of By Znode quantity is displayed. Click Modify under Operation. Increase the threshold by referringto the value of max.Znode.count by choosing Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > Quota.
- In the Used Resources (By Second-Level Znode), choose
> By capacity. The Threshold Settings page of By Capacity is displayed. Click Modify under Operation. Increase the threshold by referring to the value of max.data.size by choosing Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > Quota.
- Check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-13007.html b/docs/mrs/umn/ALM-13007.html index 08e93b53..2c10aa76 100644 --- a/docs/mrs/umn/ALM-13007.html +++ b/docs/mrs/umn/ALM-13007.html @@ -75,10 +75,10 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Procedure
Check whether there are a large number of client processes connected to ZooKeeper.
- On FusionInsight Manager, choose O&M > Alarm > Alarms. On the displayed interface, click the drop-down button of Available ZooKeeper Client Connections Are Insufficient. Confirm the node IP address of the host for which the alarm is generated in the Location Information.
- Open the ZooKeeper service interface, click Resource to enter the Resource page, and check whether the number of connections of the client with the IP address specified by Number of Connections (By Client IP Address) is large. -
- Check whether connection leakage occurs on the client process.
- Click
in the Number of Connections (by Client IP Address) to enter the Thresholds page, and click Modify under Operation. Increase the threshold by referring to the value of maxClientCnxns by choosing Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > quorumpeer.
- Check whether the alarm is cleared.
+
- If it is, no further action is required.
- If it is not, go to 6.
- Check whether connection leakage occurs on the client process.
- Click
in the Number of Connections (by Client IP Address) to enter the Thresholds page, and click Modify under Operation. Increase the threshold by referring to the value of maxClientCnxns by choosing Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > quorumpeer.
- Check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 6.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-13008.html b/docs/mrs/umn/ALM-13008.html index 48500587..3cfb7667 100644 --- a/docs/mrs/umn/ALM-13008.html +++ b/docs/mrs/umn/ALM-13008.html @@ -68,7 +68,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, select Location from the drop-down list box next to ALM-13008 ZooKeeper Znode Quantity Usage Exceeds Threshold, and obtain the Znode path in ServiceDirectory.
- Log in to the ZooKeeper client as a cluster user and delete unnecessary data from the Znode corresponding to the alarm.
- Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations, and search for max.znode.count, which is the maximum number of ZooKeeper directories. The alarm threshold is 80% of this parameter. Increase the value of this parameter, click Save, and restart the service for the configuration to take effect.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-13009.html b/docs/mrs/umn/ALM-13009.html index 289e9093..e48c325c 100644 --- a/docs/mrs/umn/ALM-13009.html +++ b/docs/mrs/umn/ALM-13009.html @@ -77,7 +77,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Log in to FusionInsight Manager and choose Cluster > Services > ZooKeeper. On the page that is displayed, click the Configuration tab then the All Configurations sub-tab, and search for max.data.size. The value of max.data.size is the maximum capacity quota of the ZooKeeper directory. The unit is byte. Search for the GC_OPTS configuration item and check the value of Xmx.
- Compare the values of max.data.size and Xmx*0.65. The threshold is the smaller value multiplied by 80%. You can change the values of max.data.size and Xmx*0.65 to increase the threshold.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ZooKeeper for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ZooKeeper for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-13010.html b/docs/mrs/umn/ALM-13010.html index b014851d..5fadf9f7 100644 --- a/docs/mrs/umn/ALM-13010.html +++ b/docs/mrs/umn/ALM-13010.html @@ -64,13 +64,13 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
Possible Causes
- A large amount of data is written to the ZooKeeper data directory.
- The user-defined threshold is inappropriate.
Procedure
Check whether a large amount of data is written into the directory for which the alarm is generated.
-
- On FusionInsight Manager, choose O&M > Alarm > Alarms. Confirm the Znode for which the alarm is generated in Location of this alarm.
- Choose Cluster > Name of the desired cluster > Services > ZooKeeper and click Resource. In Used Resources (By Second-Level Znode), check whether a large amount of data is written into the top Znode. -
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, select Location from the drop-down list box next to ALM-13010 Znode Usage of a Directory with Quota Configured Exceeds the Threshold, and obtain the Znode path in ServiceDirectory.
- Log in to the ZooKeeper client as a cluster user and delete unwanted data in the Znode for which the alarm is generated.
- Log in to FusionInsight Manager, and choose Cluster > Name of the desired cluster > Services > Component of the top Znode for which the alarm isgenerated. Choose Configurations > All Configurations, search for zk.quota.number, increase its value, click Save.
If the Component of the top Znode for which the alarm isgenerated is ClickHouse, change the value of clickhouse.zookeeper.quota.node.count.
+
- On FusionInsight Manager, choose O&M > Alarm > Alarms. Confirm the Znode for which the alarm is generated in Location of this alarm.
- Choose Cluster > Name of the desired cluster > Services > ZooKeeper and click Resource. In Used Resources (By Second-Level Znode), check whether a large amount of data is written into the top Znode. +
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, select Location from the drop-down list box next to ALM-13010 Znode Usage of a Directory with Quota Configured Exceeds the Threshold, and obtain the Znode path in ServiceDirectory.
- Log in to the ZooKeeper client as a cluster user and delete unwanted data in the Znode for which the alarm is generated.
- Log in to FusionInsight Manager, and choose Cluster > Name of the desired cluster > Services > Component of the top Znode for which the alarm isgenerated. Choose Configurations > All Configurations, search for zk.quota.number, increase its value, click Save.
If the Component of the top Znode for which the alarm isgenerated is ClickHouse, change the value of clickhouse.zookeeper.quota.node.count.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ZooKeeper in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14000.html b/docs/mrs/umn/ALM-14000.html index 0f9018d4..76b8dea1 100644 --- a/docs/mrs/umn/ALM-14000.html +++ b/docs/mrs/umn/ALM-14000.html @@ -71,7 +71,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service:
-
- ZooKeeper
- HDFS
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14001.html b/docs/mrs/umn/ALM-14001.html index 74a6f301..6e3ed0d7 100644 --- a/docs/mrs/umn/ALM-14001.html +++ b/docs/mrs/umn/ALM-14001.html @@ -82,7 +82,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service:
-
- ZooKeeper
- HDFS
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14002.html b/docs/mrs/umn/ALM-14002.html index 57b39daa..9affcf15 100644 --- a/docs/mrs/umn/ALM-14002.html +++ b/docs/mrs/umn/ALM-14002.html @@ -78,7 +78,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Wait several minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-14003.html b/docs/mrs/umn/ALM-14003.html index 61bf15fd..e77b9c30 100644 --- a/docs/mrs/umn/ALM-14003.html +++ b/docs/mrs/umn/ALM-14003.html @@ -66,7 +66,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
Impact on the System
Data stored in HDFS is lost. HDFS may enter the safe mode and cannot provide write services. Lost block data cannot be restored.
+Impact on the System
Data stored in HDFS is lost. HDFS may enter the security mode and cannot provide write services. Lost block data cannot be restored.
@@ -94,7 +94,7 @@Possible Causes
- The DataNode instance is abnormal.
- Data is deleted.
- Choose O&M > Alarm > Alarms and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-14006.html b/docs/mrs/umn/ALM-14006.html index a436262b..edeccf0a 100644 --- a/docs/mrs/umn/ALM-14006.html +++ b/docs/mrs/umn/ALM-14006.html @@ -81,7 +81,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- If yes, no further action is required.
- If no, go to 9.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-14007.html b/docs/mrs/umn/ALM-14007.html index c170b7a2..8e4fefc6 100644 --- a/docs/mrs/umn/ALM-14007.html +++ b/docs/mrs/umn/ALM-14007.html @@ -83,7 +83,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service:
-
- ZooKeeper
- HDFS
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14008.html b/docs/mrs/umn/ALM-14008.html index 7d7d3013..2b13520a 100644 --- a/docs/mrs/umn/ALM-14008.html +++ b/docs/mrs/umn/ALM-14008.html @@ -82,7 +82,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Modify the heap memory parameters of the DataNode based on the mapping between the number of blocks and the memory. Click Save and choose Dashboard > More > Restart Service.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14009.html b/docs/mrs/umn/ALM-14009.html index 7606d4d1..dd1e28e4 100644 --- a/docs/mrs/umn/ALM-14009.html +++ b/docs/mrs/umn/ALM-14009.html @@ -95,7 +95,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Check whether the alarm is cleared from the alarm list.
- If yes, no further action is required.
- If no, go to 16.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14010.html b/docs/mrs/umn/ALM-14010.html index bb625feb..b6da6063 100644 --- a/docs/mrs/umn/ALM-14010.html +++ b/docs/mrs/umn/ALM-14010.html @@ -92,7 +92,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect the fault information.
+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- In the Service area, select the following nodes of the desired cluster.
-
- ZooKeeper
- HDFS
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-14011.html b/docs/mrs/umn/ALM-14011.html index c44f80b4..153fb431 100644 --- a/docs/mrs/umn/ALM-14011.html +++ b/docs/mrs/umn/ALM-14011.html @@ -83,7 +83,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS > Instance and restart the DataNode instance
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 26.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14012.html b/docs/mrs/umn/ALM-14012.html index da137c54..7be9951e 100644 --- a/docs/mrs/umn/ALM-14012.html +++ b/docs/mrs/umn/ALM-14012.html @@ -79,7 +79,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Contact the network administrator to rectify the network fault and check whether the alarm is cleared 5 minutes later.
- If yes, no further action is required.
- If no, go to 15.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14013.html b/docs/mrs/umn/ALM-14013.html index 99b85ca9..20ebe2b8 100644 --- a/docs/mrs/umn/ALM-14013.html +++ b/docs/mrs/umn/ALM-14013.html @@ -99,7 +99,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Clear the redundant files on the disk where the directory resides to reserve sufficient space for metadata. After the clearance, wait for a NameNode metadata combination period and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 30.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NameNode in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NameNode in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14014.html b/docs/mrs/umn/ALM-14014.html index 073597fe..d4e366e7 100644 --- a/docs/mrs/umn/ALM-14014.html +++ b/docs/mrs/umn/ALM-14014.html @@ -72,7 +72,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Save the configuration and restart the NameNode instance.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NameNode in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NameNode in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14015.html b/docs/mrs/umn/ALM-14015.html index 73f03f50..eb3a147f 100644 --- a/docs/mrs/umn/ALM-14015.html +++ b/docs/mrs/umn/ALM-14015.html @@ -72,7 +72,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Save the configuration and restart the DataNode instance.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select DataNode in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select DataNode in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14016.html b/docs/mrs/umn/ALM-14016.html index 6fa0c62c..1b8bdbfb 100644 --- a/docs/mrs/umn/ALM-14016.html +++ b/docs/mrs/umn/ALM-14016.html @@ -80,7 +80,7 @@ Mem: 112 48Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select DataNode for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select DataNode for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-14017.html b/docs/mrs/umn/ALM-14017.html index 4157b778..c709e773 100644 --- a/docs/mrs/umn/ALM-14017.html +++ b/docs/mrs/umn/ALM-14017.html @@ -71,7 +71,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NameNode in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NameNode in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14018.html b/docs/mrs/umn/ALM-14018.html index 0246e8c5..1d9372f0 100644 --- a/docs/mrs/umn/ALM-14018.html +++ b/docs/mrs/umn/ALM-14018.html @@ -83,7 +83,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following services in the required cluster from the Service.
-
- ZooKeeper
- HDFS
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14019.html b/docs/mrs/umn/ALM-14019.html index 8e171e9c..278555a2 100644 --- a/docs/mrs/umn/ALM-14019.html +++ b/docs/mrs/umn/ALM-14019.html @@ -83,7 +83,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following services in the required cluster from the Service.
-
- ZooKeeper
- HDFS
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14020.html b/docs/mrs/umn/ALM-14020.html index d9b61f0a..120a298e 100644 --- a/docs/mrs/umn/ALM-14020.html +++ b/docs/mrs/umn/ALM-14020.html @@ -86,7 +86,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Increase the parameter value.
- Save the configuration and click Dashboard > More > Restart Service.
- Wait 1 hour and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14021.html b/docs/mrs/umn/ALM-14021.html index 255cd25b..84d6d735 100644 --- a/docs/mrs/umn/ALM-14021.html +++ b/docs/mrs/umn/ALM-14021.html @@ -90,7 +90,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following node in the required cluster from the Service.
-
- HDFS
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14022.html b/docs/mrs/umn/ALM-14022.html index 63218b80..0af9b05b 100644 --- a/docs/mrs/umn/ALM-14022.html +++ b/docs/mrs/umn/ALM-14022.html @@ -87,11 +87,11 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS > Configurations > All Configurations. Search for parameter dfs.namenode.handler.count and view its value. If the value is less than or equal to 128, change it to 128. If the value is greater than 128 but less than 192, change it to 192.
- Search for parameter ipc.server.read.threadpool.size and view its value. If the value is less than 5, change it to 5.
- Click Save, and click OK.
- On the Instance page of HDFS, select the standby NameNode of NameService involved in this alarm and choose More > Restart Instance. Enter the password and click OK. Wait until the standby NameNode is started up.
- On the Instance page of HDFS, select the active NameNode of NameService involved in this alarm and choose More > Restart Instance. Enter the password and click OK. Wait until the active NameNode is started up.
- Wait for 1 hour and then check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 20.
Check whether the HDFS workload changes and reduce the workload properly.
-
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS. Click the drop-down menu in the upper right corner of Chart, click Customize, select Average Time of Active NameNode RPC Queuing and click OK.
- Click
. The Details page is displayed.
- Set the monitoring data display period, from 5 days before the alarm generation time to the alarm generation time. Click OK.
- On the Average RPC Queuing Time monitoring page, check whether the point in time when the queuing time increases abruptly exists. +
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS. Click the drop-down menu in the upper right corner of Chart, click Customize, select Average Time of Active NameNode RPC Queuing and click OK.
- Click
. The Details page is displayed.
- Set the monitoring data display period, from 5 days before the alarm generation time to the alarm generation time. Click OK.
- On the Average RPC Queuing Time monitoring page, check whether the point in time when the queuing time increases abruptly exists.
- Confirm and check the point in time. Check whether a new task frequently accesses HDFS and whether the access frequency can be reduced.
- If a Balancer task starts at the point in time, stop the task or specify a node for the task to reduce the HDFS workload.
- Wait for 1 hour and then check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 27.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14023.html b/docs/mrs/umn/ALM-14023.html index 5ced95d7..c222e393 100644 --- a/docs/mrs/umn/ALM-14023.html +++ b/docs/mrs/umn/ALM-14023.html @@ -77,7 +77,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Expand the DataNode capacity.
- Wait 5 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14024.html b/docs/mrs/umn/ALM-14024.html index e8f37e3b..739dfc97 100644 --- a/docs/mrs/umn/ALM-14024.html +++ b/docs/mrs/umn/ALM-14024.html @@ -83,7 +83,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- On the Resources page, click Modify to modify the storage space quota.
- About one minute later, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster and NodeAgent under Manager from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster and NodeAgent under Manager from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14025.html b/docs/mrs/umn/ALM-14025.html index 12ab4d73..27875115 100644 --- a/docs/mrs/umn/ALM-14025.html +++ b/docs/mrs/umn/ALM-14025.html @@ -83,7 +83,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- On the Resources page, click Modify to modify or delete the maximum number of file objects configured for the tenant directory.
- About one minute later, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster and NodeAgent under Manager from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HDFS in the required cluster and NodeAgent under Manager from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-14026.html b/docs/mrs/umn/ALM-14026.html index 8917c296..a08d81ae 100644 --- a/docs/mrs/umn/ALM-14026.html +++ b/docs/mrs/umn/ALM-14026.html @@ -77,7 +77,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Expand the DataNode capacity.
- On FusionInsight Manager, wait for 5 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 10.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-14027.html b/docs/mrs/umn/ALM-14027.html index d64dc9f8..af32ed36 100644 --- a/docs/mrs/umn/ALM-14027.html +++ b/docs/mrs/umn/ALM-14027.html @@ -76,7 +76,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- In the alarm list on Manager, click Clear in the Operation column of the alarm to manually clear the alarm. Choose Cluster > Services > HDFS > Instance, select the DataNode, choose More > Restart Instance, wait for 5 minutes, and check whether a new alarm is reported.
- If no, no further action is required.
- If yes, go to 8.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HDFS and OMS for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HDFS and OMS for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-14028.html b/docs/mrs/umn/ALM-14028.html index c385aa95..a819f294 100644 --- a/docs/mrs/umn/ALM-14028.html +++ b/docs/mrs/umn/ALM-14028.html @@ -92,7 +92,7 @@Alarm Clearing
After the fault is rectified, the system does not automatically clear this alarm and you need to manually clear the alarm.
- If yes, no further action is required.
- If no, go to 9.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-14029.html b/docs/mrs/umn/ALM-14029.html index e0176c52..dee09547 100644 --- a/docs/mrs/umn/ALM-14029.html +++ b/docs/mrs/umn/ALM-14029.html @@ -78,7 +78,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- In the next detection period, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-14030.html b/docs/mrs/umn/ALM-14030.html index 19a09cb0..00a07d1b 100644 --- a/docs/mrs/umn/ALM-14030.html +++ b/docs/mrs/umn/ALM-14030.html @@ -50,14 +50,14 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
Impact on the System
Data of a single replica may be lost. Therefore, the system does not allow write of single-replica data by default. If this configuration is enabled on HDFS and the number of HDFS replicas configured on the client is 1, single-replica data can be written to HDFS.
+Impact on the System
If this configuration is enabled on the server and the number of HDFS replicas configured on the client is 1, single-replica data can be written to HDFS. Data of a single replica may be lost. Therefore, the system does not allow write of single-replica data by default. If a service requires single-replica data write to a directory, modify the HDFS configuration item dfs.single.replication.exclude.pattern.
-Possible Causes
The HDFS configuration item dfs.single.replication.enable is set to true.
Procedure
- Log in to FusionInsight Manager and choose Cluster > Services > HDFS. On the page that is displayed, click the Configurations tab then the All Configurations sub-tab.
- Search for dfs.single.replication.enable in the search box, change the value of the configuration item to false, and click Save.
- On the Dashboard page of the HDFS service, click More and select Service Rolling Restart in the upper right corner.
- After the HDFS service is started, check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 5.
Procedure
- Log in to FusionInsight Manager and choose Cluster > Services > HDFS. On the page that is displayed, click the Configurations tab then the All Configurations sub-tab.
- Search for dfs.single.replication.enable in the search box, change the value of the configuration item to false, and click Save.
- Wait for about 10 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-16000.html b/docs/mrs/umn/ALM-16000.html index 5f8313ce..4cade166 100644 --- a/docs/mrs/umn/ALM-16000.html +++ b/docs/mrs/umn/ALM-16000.html @@ -69,7 +69,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- On the FusionInsight Manager portal, Choose Cluster > Name of the desired cluster > Services > Hive > Configurations >All Configurations.
- Search for hive.server.session.control.maxconnections and increase the value of this parameter. If the value of this parameter is A, the threshold is B, and the number of sessions connected to the HiveServer is C, adjust the value of this parameter according to A x B > C. To view the number of sessions connected to the HiveServer, check the value of Statistics for Sessions of the HiveServer on the Hive monitoring page.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-16001.html b/docs/mrs/umn/ALM-16001.html index 67c739c2..64d7a85e 100644 --- a/docs/mrs/umn/ALM-16001.html +++ b/docs/mrs/umn/ALM-16001.html @@ -79,7 +79,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Clear the alarm by following the steps provided in "ALM-12006 Node Fault", "ALM-12007 Process Fault", and "ALM-14002 DataNode Disk Usage Exceeds the Threshold".
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-16002.html b/docs/mrs/umn/ALM-16002.html index f652f476..276d8f88 100644 --- a/docs/mrs/umn/ALM-16002.html +++ b/docs/mrs/umn/ALM-16002.html @@ -83,7 +83,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager home page, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service:
-
- MapReduce
- Hive
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-16003.html b/docs/mrs/umn/ALM-16003.html index cebcd4ed..1663d62e 100644 --- a/docs/mrs/umn/ALM-16003.html +++ b/docs/mrs/umn/ALM-16003.html @@ -2,8 +2,6 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-16003 Background Thread Usage Exceeds the Threshold
Description
The system checks the background thread usage in every 30 seconds. This alarm is generated when the usage of the background thread pool of Hive exceeds the threshold, 90% by default.
-MRS 3.X supports the multi-instance function. If the multi-instance function is enabled in the cluster and multiple Hive services are installed, determine the Hive service for which the alarm is generated based on the value of ServiceName in Location of the alarm. For example, if Hive1 service is unavailable, ServiceName is set to Hive1 in Location, and the operation object in the handling procedure is changed from Hive to Hive1.
-Attribute
Alarm ID
@@ -74,7 +72,7 @@
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Hive. On the displayed page, click HiveServer Instance and check values of Background Thread Count and Background Thread Usage.
- Increase the value of hive.server2.async.exec.threads in the ${BIGDATA_HOME}/FusionInsight_HD_8.1.0.1/1_23_HiveServer/etc/hive-site.xml file. For example, increase the value by 20%.
- Save the modification.
- Check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 9.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-16004.html b/docs/mrs/umn/ALM-16004.html index 3ba2381e..91824b35 100644 --- a/docs/mrs/umn/ALM-16004.html +++ b/docs/mrs/umn/ALM-16004.html @@ -3,8 +3,6 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-16004 Hive Service Unavailable
Description
This alarm is generated when the HiveServer service is unavailable. The system checks the HiveServer service status every 60 seconds.
This alarm is cleared when the HiveServer service is normal.
-MRS 3.X supports the multi-instance function. If the multi-instance function is enabled in the cluster and multiple Hive service instances are installed, you need to determine the Hive service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the Hive1 service is unavailable, ServiceName=Hive1 is displayed in Location, and the operation object in the procedure needs to be changed from Hive to Hive1.
-Attribute
Alarm ID
@@ -91,7 +89,7 @@Collect fault information.
+
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service:
-
- ZooKeeper
- HDFS
- Yarn
- DBService
- Hive
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-16005.html b/docs/mrs/umn/ALM-16005.html index 9a2dc931..97204a36 100644 --- a/docs/mrs/umn/ALM-16005.html +++ b/docs/mrs/umn/ALM-16005.html @@ -73,7 +73,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Click More > Restart Service to restart the service.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-16006.html b/docs/mrs/umn/ALM-16006.html index 859cfe70..353179f9 100644 --- a/docs/mrs/umn/ALM-16006.html +++ b/docs/mrs/umn/ALM-16006.html @@ -77,7 +77,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Click More > Restart Service to restart the service.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
diff --git a/docs/mrs/umn/ALM-16007.html b/docs/mrs/umn/ALM-16007.html index dc818da6..51772d70 100644 --- a/docs/mrs/umn/ALM-16007.html +++ b/docs/mrs/umn/ALM-16007.html @@ -79,7 +79,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Click More > Restart Service to restart the service.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal of active and standby clusters, choose O&M > Log > Download.
- In the Service, select Hive in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal of active and standby clusters, choose O&M > Log > Download.
- In the Service, select Hive in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-16008.html b/docs/mrs/umn/ALM-16008.html index 9a7c98fa..a5436293 100644 --- a/docs/mrs/umn/ALM-16008.html +++ b/docs/mrs/umn/ALM-16008.html @@ -74,7 +74,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Click More > Restart Service to restart the service.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-16009.html b/docs/mrs/umn/ALM-16009.html index 6f8c1105..d7fffe70 100644 --- a/docs/mrs/umn/ALM-16009.html +++ b/docs/mrs/umn/ALM-16009.html @@ -67,7 +67,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- On FusionInsight Manager portal, choose Cluster >Name of the desired cluster > Services > Hive > Resource. Check the HQL statements with the excessively large number (5000 or more) of maps in HQL Map Count.
- Locate the corresponding HQL statements, optimize them and execute them again.
- Check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Collect fault information.
-+
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select Hive in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-16047.html b/docs/mrs/umn/ALM-16047.html index 8fda4a06..71bff408 100644 --- a/docs/mrs/umn/ALM-16047.html +++ b/docs/mrs/umn/ALM-16047.html @@ -62,7 +62,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Log in to FusionInsight Manager. Choose O&M > Alarm > Alarms, click the drop-down list in the row that contains the alarm, and view role and the IP address of the node for which the alarm is generated in Location.
- Choose Cluster > Name of the desired cluster > Services > Hive > Instance, select the instance at the IP address for which the alarm is generated, and choose More > Restart Instance.
- Wait for 5 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Hive for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Hive for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-17004.html b/docs/mrs/umn/ALM-17004.html index ccee4d6b..b1da6efa 100644 --- a/docs/mrs/umn/ALM-17004.html +++ b/docs/mrs/umn/ALM-17004.html @@ -71,7 +71,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Oozie in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Oozie in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-17005.html b/docs/mrs/umn/ALM-17005.html index 359fc8cb..278ba83b 100644 --- a/docs/mrs/umn/ALM-17005.html +++ b/docs/mrs/umn/ALM-17005.html @@ -72,7 +72,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Oozie for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Oozie for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-17006.html b/docs/mrs/umn/ALM-17006.html index 48fec24d..616bc35c 100644 --- a/docs/mrs/umn/ALM-17006.html +++ b/docs/mrs/umn/ALM-17006.html @@ -71,7 +71,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Oozie in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Oozie in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-17007.html b/docs/mrs/umn/ALM-17007.html index 5599a734..fb9b20fe 100644 --- a/docs/mrs/umn/ALM-17007.html +++ b/docs/mrs/umn/ALM-17007.html @@ -71,7 +71,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Oozie in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Oozie in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18000.html b/docs/mrs/umn/ALM-18000.html index 172c3be3..3621f3a6 100644 --- a/docs/mrs/umn/ALM-18000.html +++ b/docs/mrs/umn/ALM-18000.html @@ -75,7 +75,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Rectify the fault by following the steps provided in ALM-18002 NodeManager Heartbeat Lost or ALM-18003 NodeManager Unhealthy. After the fault is rectified, check whether the Yarn alarm is cleared.
- If yes, no further action is required.
- If no, go to 10.
Collect fault information.
-+
- On the FusionInsight Manager portal of the active cluster, choose O&M > Log > Download.
- Select Yarn in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal of the active cluster, choose O&M > Log > Download.
- Select Yarn in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18002.html b/docs/mrs/umn/ALM-18002.html index 5fdec8e7..9f3a5211 100644 --- a/docs/mrs/umn/ALM-18002.html +++ b/docs/mrs/umn/ALM-18002.html @@ -67,7 +67,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Possible Causes
- NodeManager is forcibly deleted without decommission.
- All the NodeManager instances are stopped or the NodeManager process is faulty.
- The host where the NodeManager node resides is faulty.
- The network between the NodeManager and ResourceManager is disconnected or busy.
Procedure
Check the NodeManager status.
-
- On the FusionInsight Manager, and choose O&M > Alarm > Alarms. Click
before the alarm and obtain lost nodes in Additional Information.
- Check whether the lost nodes are hosts that have been manually deleted without decommission. +
- On the FusionInsight Manager, and choose O&M > Alarm > Alarms. Click
before the alarm and obtain lost nodes in Additional Information.
- Check whether the lost nodes are hosts that have been manually deleted without decommission.
- After the setting, Choose Cluster > Name of the desired cluster > Services > Yarn. On the displayed page, choose Configurations > All Configurations. Search for yarn.nodemanager.lost.alarm.threshold and change its value to the number of hosts that are not out of service and proactively deleted. After the setting, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
- Manually clear the alarm. Note that decommission must be performed before deleting hosts.
- On the FusionInsight Manager portal, choose Cluster > Hosts, and check whether the nodes obtained in 1 are healthy.
- Rectify the node fault based on ALM-12006 Node Fault and check whether the alarm is cleared.
@@ -85,7 +85,7 @@
- If yes, no further action is required.
- If no, go to 7.
- Rectify the network, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 13.
Collect fault information.
-+
- On the FusionInsight Manager in the active cluster, choose O&M > Log > Download.
- Select Yarn in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager in the active cluster, choose O&M > Log > Download.
- Select Yarn in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18003.html b/docs/mrs/umn/ALM-18003.html index 507b6472..513c5e82 100644 --- a/docs/mrs/umn/ALM-18003.html +++ b/docs/mrs/umn/ALM-18003.html @@ -66,7 +66,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Possible Causes
- The hard disk space of the host where the NodeManager node resides is insufficient.
- User omm does not have the permission to access a local directory on the NodeManager node.
Procedure
Check the hard disk space of the host.
-
- On the FusionInsight Manager, and choose O&M > Alarm > Alarms. Click
before the alarm and obtain unhealthy nodes in Additional Information.
- Choose Cluster > Name of the desired cluster > Services > Yarn > Instance, select the NodeManager instance corresponding to the host, choose Instance Configurations > All Configurations and view disks corresponding to yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs.
- Choose O&M > Alarm > Alarms. In the alarm list, check whether the related disk has the alarm ALM-12017 Insufficient Disk Capacity. +
- On the FusionInsight Manager, and choose O&M > Alarm > Alarms. Click
before the alarm and obtain unhealthy nodes in Additional Information.
- Choose Cluster > Name of the desired cluster > Services > Yarn > Instance, select the NodeManager instance corresponding to the host, choose Instance Configurations > All Configurations and view disks corresponding to yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs.
- Choose O&M > Alarm > Alarms. In the alarm list, check whether the related disk has the alarm ALM-12017 Insufficient Disk Capacity.
- Rectify the disk fault based on ALM-12017 Insufficient Disk Capacit and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
- Choose Hosts > Name of the desired host . On the Dashboard page, check the disk usage of the corresponding partition. Check whether the percentage of the used space of the mounted disk exceeds the value of yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage
- Reduce the disk usage to less than the value of yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage, wait for 10 to 20 minutes, and check whether the alarm is cleared.
@@ -78,7 +78,7 @@
- If yes, no further action is required.
- If no, go to 7.
- Wait for 10 to 20 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.
Collect fault information.
-+
- On the FusionInsight Manager in the active cluster, choose O&M > Log > Download.
- Select Yarn in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager in the active cluster, choose O&M > Log > Download.
- Select Yarn in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18008.html b/docs/mrs/umn/ALM-18008.html index bbe47f77..a1d3e791 100644 --- a/docs/mrs/umn/ALM-18008.html +++ b/docs/mrs/umn/ALM-18008.html @@ -74,7 +74,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following node in the required cluster from the Service.
-
- NodeAgent
- Yarn
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18009.html b/docs/mrs/umn/ALM-18009.html index 55db4644..9a04d76c 100644 --- a/docs/mrs/umn/ALM-18009.html +++ b/docs/mrs/umn/ALM-18009.html @@ -74,7 +74,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following node in the required cluster from the Service.
-
- NodeAgent
- Mapreduce
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18010.html b/docs/mrs/umn/ALM-18010.html index 08503139..10070b3f 100644 --- a/docs/mrs/umn/ALM-18010.html +++ b/docs/mrs/umn/ALM-18010.html @@ -72,7 +72,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Save the configuration and restart the ResourceManager instance.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ResourceManager in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ResourceManager in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18011.html b/docs/mrs/umn/ALM-18011.html index aa8394f2..832b3601 100644 --- a/docs/mrs/umn/ALM-18011.html +++ b/docs/mrs/umn/ALM-18011.html @@ -72,7 +72,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Save the configuration and restart the NodeManager instance.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NodeManager in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NodeManager in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18012.html b/docs/mrs/umn/ALM-18012.html index ff36740e..38749f7d 100644 --- a/docs/mrs/umn/ALM-18012.html +++ b/docs/mrs/umn/ALM-18012.html @@ -72,7 +72,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Save the configuration and restart the JobHistoryServer instance.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select JobHistoryServer in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select JobHistoryServer in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18013.html b/docs/mrs/umn/ALM-18013.html index f70c5148..d4e5d23b 100644 --- a/docs/mrs/umn/ALM-18013.html +++ b/docs/mrs/umn/ALM-18013.html @@ -71,7 +71,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ResourceManager in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select ResourceManager in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18014.html b/docs/mrs/umn/ALM-18014.html index 7c2dd4b4..3d5e4fd6 100644 --- a/docs/mrs/umn/ALM-18014.html +++ b/docs/mrs/umn/ALM-18014.html @@ -71,7 +71,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NodeManager in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select NodeManager in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18015.html b/docs/mrs/umn/ALM-18015.html index ac44a569..975303c8 100644 --- a/docs/mrs/umn/ALM-18015.html +++ b/docs/mrs/umn/ALM-18015.html @@ -71,7 +71,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select JobHistoryServer in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select JobHistoryServer in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18016.html b/docs/mrs/umn/ALM-18016.html index 120023e4..b000ebfc 100644 --- a/docs/mrs/umn/ALM-18016.html +++ b/docs/mrs/umn/ALM-18016.html @@ -74,7 +74,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following node in the required cluster from the Service.
-
- NodeAgent
- Yarn
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18017.html b/docs/mrs/umn/ALM-18017.html index 36f92a82..fae66cc2 100644 --- a/docs/mrs/umn/ALM-18017.html +++ b/docs/mrs/umn/ALM-18017.html @@ -74,7 +74,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following node in the required cluster from the Service.
-
- NodeAgent
- Yarn
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18018.html b/docs/mrs/umn/ALM-18018.html index 1733c818..f84d3c9c 100644 --- a/docs/mrs/umn/ALM-18018.html +++ b/docs/mrs/umn/ALM-18018.html @@ -73,7 +73,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following node in the required cluster from the Service.
-
- NodeAgent
- Yarn
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18019.html b/docs/mrs/umn/ALM-18019.html index 4f89eb4a..642b9219 100644 --- a/docs/mrs/umn/ALM-18019.html +++ b/docs/mrs/umn/ALM-18019.html @@ -74,7 +74,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following node in the required cluster from the Service.
-
- NodeAgent
- MapReduce
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18020.html b/docs/mrs/umn/ALM-18020.html index 817ca127..40801cd0 100644 --- a/docs/mrs/umn/ALM-18020.html +++ b/docs/mrs/umn/ALM-18020.html @@ -65,19 +65,19 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Possible Causes
- The specified timeout duration is shorter than the required execution time.
- The queue resources for task running are insufficient.
- Task data skew occurs. As a result, some tasks process a large amount of data and take a long time to execute.
Procedure
Check whether the timeout interval is correctly set.
-
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Alarm > Alarms. The Alarms page is displayed.
- Select the alarm whose ID is 18020. In the alarm details, view Location to obtain the timeout task name and timeout duration.
- Based on the task name and timeout interval, choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager (Active) to log in to the native Yarn page. Then find the task on the native page, check its StartTime and calculate the task execution time based on the current system time. Check whether the task execution time exceeds the timeout duration. -
- Evaluate the expected task execution time based on the service and compare it with the task timeout interval. If the timeout interval is too short, set the timeout interval (mapreduce.application.timeout.alarm or spark.application.timeout.alarm) of the client to the task expected execution time. Run the task again and check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 5.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Alarm > Alarms. The Alarms page is displayed.
- Select the alarm whose ID is 18020. In the alarm details, view Location to obtain the timeout task name and timeout duration.
- Based on the task name and timeout interval, choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager (Active) to log in to the native Yarn page. Then find the task on the native page, check its StartTime and calculate the task execution time based on the current system time. Check whether the task execution time exceeds the timeout duration. +
- Evaluate the expected task execution time based on the service and compare it with the task timeout interval. If the timeout interval is too short, set the timeout interval (mapreduce.application.timeout.alarm or spark.application.timeout.alarm) of the client to the task expected execution time. Run the task again and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check whether the queue resources are sufficient.
-
- Find the task on the native page and view the queue name of the task. Click Scheduler on the left of the native page. On the Applications Queues page, find the corresponding queue name and expand the queue details, as shown in the following figure.
+
- Find the task on the native page and view the queue name of the task. Click Scheduler on the left of the native page. On the Applications Queues page, find the corresponding queue name and expand the queue details, as shown in the following figure.
- Check whether the value of Used Resources in the queue details is approximately equal to the value of Max Resources, which indicates that the resources in the queue submitted by the task have been used up. If the queue resources are insufficient, choose Tenant Resources > Dynamic Resource Plan > Resource Distribution Policy on FusionInsight Manager and increase the value of Max Resources for the queue. Run the task again and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Check whether data skew occurs.
-
- On the native Yarn page, click task ID (for example, application_1565337919723_0002) > Tracking URL:ApplicationMaster > job_1565337919723_0002. The following page is displayed.
+
- On the native Yarn page, click task ID (for example, application_1565337919723_0002) > Tracking URL:ApplicationMaster > job_1565337919723_0002. The following page is displayed.
- Choose Job > Map tasks or Job > Reduce tasks on the left and check whether the execution time of each Map or Reduce task differs greatly. If yes, task data skew occurs. In this case, you need to balance the task data.
- Rectify the fault based on the preceding causes and perform the tasks again. Then, check whether the alarm persists.
- If yes, go to 10.
- If no, no further action is required.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-18021.html b/docs/mrs/umn/ALM-18021.html index 75d279d7..9a8c892e 100644 --- a/docs/mrs/umn/ALM-18021.html +++ b/docs/mrs/umn/ALM-18021.html @@ -79,7 +79,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Rectify the fault by following the steps provided in ALM-18000 Yarn Service Unavailable, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.
Collect fault information.
-+
- On the FusionInsight Manager home page of the active cluster, choose O&M > Log > Download.
- Select MapReduce in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager home page of the active cluster, choose O&M > Log > Download.
- Select MapReduce in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18022.html b/docs/mrs/umn/ALM-18022.html index 0f9f06c9..5f43e515 100644 --- a/docs/mrs/umn/ALM-18022.html +++ b/docs/mrs/umn/ALM-18022.html @@ -75,7 +75,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
-+
- Log in to FusionInsight Manager of the active cluster, and choose O&M > Log > Download.
- Select Yarn in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Log in to FusionInsight Manager of the active cluster, and choose O&M > Log > Download.
- Select Yarn in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18023.html b/docs/mrs/umn/ALM-18023.html index d12fbbe3..4d2b22d2 100644 --- a/docs/mrs/umn/ALM-18023.html +++ b/docs/mrs/umn/ALM-18023.html @@ -65,7 +65,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- On FusionInsight Manager, choose O&M > Alarm > Thresholds > Name of the desired cluster > Yarn > Applications > Pending Applications, and increase the thresholds as required.
- Check whether the alarm is cleared 5 minutes later.
- If yes, no further action is required.
- If no, go to 8.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-18024.html b/docs/mrs/umn/ALM-18024.html index 09ca1f2d..09d68b73 100644 --- a/docs/mrs/umn/ALM-18024.html +++ b/docs/mrs/umn/ALM-18024.html @@ -65,7 +65,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- On FusionInsight Manager, choose O&M > Alarm > Thresholds > Name of the desired cluster > Yarn > CPU and Memory > Pending Memory, and increase the threshold as required.
- Check whether the alarm is cleared 5 minutes later.
- If yes, no further action is required.
- If no, go to 8.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-18025.html b/docs/mrs/umn/ALM-18025.html index 514289be..aad40a1f 100644 --- a/docs/mrs/umn/ALM-18025.html +++ b/docs/mrs/umn/ALM-18025.html @@ -68,7 +68,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Choose O&M > Alarm > Thresholds > Name of the desired cluster > Yarn > Other > Terminated Applications of root queue to modify the threshold. Go to 6.
- Choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager(Active) to access the ResourceManager web UI.
- Click KILLED in Applications and click the task on the top. View the description of Diagnostics and rectify the fault based on the task termination details (for example, the task is terminated by a user).
- Wait for 3 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect the fault information.
-+
- On the FusionInsight Manager, choose O&M > Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager, choose O&M > Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-18026.html b/docs/mrs/umn/ALM-18026.html index 2e37ea9f..d40d7077 100644 --- a/docs/mrs/umn/ALM-18026.html +++ b/docs/mrs/umn/ALM-18026.html @@ -68,7 +68,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Choose O&M > Alarm > Thresholds > Name of the desired cluster > Yarn > Other > Failed Applications of root queue to modify the threshold. Go to 6.
- Choose Cluster > Name of the desired cluster > Services > Yarn > ResourceManager(Active) to access the ResourceManager web UI.
- Click FAILED in Applications and click the task on the top. View the description of Diagnostics and rectify the fault based on the task failure causes.
- Wait for 3 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect the fault information.
-+
- On the FusionInsight Manager, choose O&M > Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager, choose O&M > Log > Download.
- Expand the Service drop-down list, and select Yarn for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-19000.html b/docs/mrs/umn/ALM-19000.html index c2ae00bf..5f57eb44 100644 --- a/docs/mrs/umn/ALM-19000.html +++ b/docs/mrs/umn/ALM-19000.html @@ -3,8 +3,6 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-19000 HBase Service Unavailable
Description
This alarm is generated when the HBase service is unavailable. The alarm module checks the HBase service status every 120 seconds.
This alarm is cleared when the HBase service recovers.
-If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the HBase1 service is unavailable, ServiceName=HBase1 is displayed in Location, and the operation object in the procedure needs to be changed from HBase to HBase1.
-Attribute
Alarm ID
@@ -81,17 +79,17 @@- Check whether at least one RegionServer exists under Region Servers.
- Check Tables > System Tables, as shown in Figure 1. Check whether hbase:meta, hbase:namespace, and hbase:acl exist in the Table Name column. - +
- As shown in Figure 1, click the hbase:meta, hbase:namespace, and hbase:acl hyperlinks and check whether the pages are properly displayed. If the pages are properly displayed, the tables are normal.
If they are, go to 19.
If they are not, go to 23.
In normal mode, ACL is enabled for HBase by default. The hbase:acl table is generated only when ACL is manually enabled. In this case, check this table. In other scenarios, this table does not need to be checked.
- View the HMaster startup status.
In Figure 2, if the RUNNING state exists in Tasks, HMaster is being started. In the State column, you can view the time when HMaster is in the RUNNING state. In Figure 3, if the state is COMPLETE, HMaster is started.
Check whether HMaster is in the RUNNING state for a long time.
- - + + -- On the HMaster WebUI, check whether any hbase:meta is in the Region in Transition state for a long time.
Figure 4 Region in Transition+- On the HMaster WebUI, check whether any hbase:meta is in the Region in Transition state for a long time.
Figure 4 Region in Transition- In the precondition that services are not affected, log in to the FusionInsight Manager portal and choose Cluster > Name of the desired cluster > Services > HBase > More > Restart Service. Enter the administrator password and click OK.
- Wait several minutes and check whether the alarm is cleared.
@@ -102,7 +100,7 @@
- If yes, no further action is required.
- If no, go to 23.
Collect fault information.
+
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service drop-down list:
-
- ZooKeeper
- HDFS
- HBase
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-19006.html b/docs/mrs/umn/ALM-19006.html index 6a46c51e..f634bee1 100644 --- a/docs/mrs/umn/ALM-19006.html +++ b/docs/mrs/umn/ALM-19006.html @@ -3,8 +3,6 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-19006 HBase Replication Sync Failed
Description
The alarm module checks the HBase DR data synchronization status every 30 seconds. When disaster recovery (DR) data fails to be synchronized to a standby cluster, the alarm is triggered.
When DR data synchronization succeeds, the alarm is cleared.
-If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the HBase1 service is unavailable, ServiceName=HBase1 is displayed in Location, and the operation object in the procedure needs to be changed from HBase to HBase1.
-Attribute
Alarm ID
@@ -93,7 +91,7 @@ abc 10.10.10.110,10.10.10.119,10.10.10.133:2181:/hbase ENABLED- Contact the network administrator to restore the network.
- After the network is running properly, check whether the alarm is cleared in the alarm list.
- If yes, no further action is required.
- If no, go to 20.
Collect fault information.
-+
- On the FusionInsight Manager interface of active and standby clusters, choose O&M > Log > Download.
- In the Service drop-down list box, select HBase in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
- On the FusionInsight Manager interface of active and standby clusters, choose O&M > Log > Download.
- In the Service drop-down list box, select HBase in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
diff --git a/docs/mrs/umn/ALM-19007.html b/docs/mrs/umn/ALM-19007.html index d8c24b93..93fac779 100644 --- a/docs/mrs/umn/ALM-19007.html +++ b/docs/mrs/umn/ALM-19007.html @@ -2,8 +2,6 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-19007 HBase GC Time Exceeds the Threshold
Description
The system checks the old generation garbage collection (GC) time of the HBase service every 60 seconds. This alarm is generated when the detected old generation GC time exceeds the threshold (exceeds 5 seconds for three consecutive checks by default). To change the threshold, on the FusionInsight Manager portal, choose O&M > Alarm > Thresholds > Name of the desired cluster > HBase > GC > GC time for old generation. This alarm is cleared when the old generation GC time of the HBase service is shorter than or equal to the threshold.
-If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the HBase1 service is unavailable, ServiceName=HBase1 is displayed in Location, and the operation object in the procedure needs to be changed from HBase to HBase1.
-Attribute
Alarm ID
@@ -73,7 +71,7 @@- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
-+
- On the FusionInsight Manager interface of active and standby clusters, choose O&M > Log > Download.
- In the Service drop-down list box, select HBase in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
- On the FusionInsight Manager interface of active and standby clusters, choose O&M > Log > Download.
- In the Service drop-down list box, select HBase in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
diff --git a/docs/mrs/umn/ALM-19008.html b/docs/mrs/umn/ALM-19008.html index 1a363259..18311246 100644 --- a/docs/mrs/umn/ALM-19008.html +++ b/docs/mrs/umn/ALM-19008.html @@ -2,8 +2,6 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-19008 Heap Memory Usage of the HBase Process Exceeds the Threshold
Description
The system checks the HBase service status every 30 seconds. The alarm is generated when the heap memory usage of an HBase service exceeds the threshold (90% of the maximum memory).
-If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the HBase1 service is unavailable, ServiceName=HBase1 is displayed in Location, and the operation object in the procedure needs to be changed from HBase to HBase1.
-Attribute
Alarm ID
@@ -71,7 +69,7 @@- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HBase in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select HBase in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
diff --git a/docs/mrs/umn/ALM-19009.html b/docs/mrs/umn/ALM-19009.html index 44f04df8..e2b6065f 100644 --- a/docs/mrs/umn/ALM-19009.html +++ b/docs/mrs/umn/ALM-19009.html @@ -3,8 +3,6 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-19009 Direct Memory Usage of the HBase Process Exceeds the Threshold
Description
The system checks the HBase service status every 30 seconds. The alarm is generated when the direct memory usage of an HBase service exceeds the threshold (90% of the maximum memory).
The alarm is cleared when the direct memory usage is less than the threshold.
-If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the HBase1 service is unavailable, ServiceName=HBase1 is displayed in Location, and the operation object in the procedure needs to be changed from HBase to HBase1.
-Attribute
Alarm ID
@@ -71,7 +69,7 @@- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect fault information.
-+
- On the FusionInsight Manager interface of active and standby clusters, choose O&M > Log > Download.
- In the Service in the required cluster drop-down list box, select HBase.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
- On the FusionInsight Manager interface of active and standby clusters, choose O&M > Log > Download.
- In the Service in the required cluster drop-down list box, select HBase.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
diff --git a/docs/mrs/umn/ALM-19011.html b/docs/mrs/umn/ALM-19011.html index 0a488386..9cefc5bc 100644 --- a/docs/mrs/umn/ALM-19011.html +++ b/docs/mrs/umn/ALM-19011.html @@ -2,8 +2,6 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-19011 RegionServer Region Number Exceeds the Threshold
Description
The system checks the number of regions on each RegionServer in each HBase service instance every 30 seconds. The region number is displayed on the HBase service monitoring page and RegionServer role monitoring page. This alarm is generated when the number of regions on a RegionServer exceeds the threshold (default value: 2000) for 20 consecutive times. The threshold can be changed by choosing O&M > Alarm > Thresholds > Name of the desired cluster > HBase. This alarm is cleared when the number of regions is less than or equal to the threshold.
-If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the HBase1 service is unavailable, ServiceName=HBase1 is displayed in Location, and the operation object in the procedure needs to be changed from HBase to HBase1.
-Attribute
Alarm ID
@@ -64,7 +62,7 @@
- On the FusionInsight Manager home page, choose O&M > Alarm > Alarms, select this alarm, and view the service instance and host name in Location.
- On the FusionInsight Manager home page, choose Cluster > Name of the desired cluster > Services, click the HBase service instance for which the alarm is generated, and click HMaster(Active). On the displayed WebUI of the HBase instance, check whether the region distribution on the RegionServer is balanced.
-By default, the admin user does not have the permissions to manage other components. If the page cannot be opened or the displayed content is incomplete when you access the native UI of a component due to insufficient permissions, you can manually create a user with the permissions to manage that component.
Figure 1 WebUI of HBase instance+Figure 1 WebUI of HBase instanceEnable load balancing.
+
- Log in to the node where the HBase client is located as user root. Go to the client installation directory, and set environment variables.
cd client installation directory
@@ -99,7 +97,7 @@- Add nodes to the HBase cluster and add RegionServer instances to the nodes. Then enable and manually trigger the load balancing function.
- On the FusionInsight Manager home page, choose Cluster > Name of the desired cluster > Services, click the HBase service instance for which the alarm is generated, and click HMaster(Active). On the displayed WebUI of the HBase instance, refresh the page and check whether the region distribution is balanced.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 21.
Collect fault information.
-- On the FusionInsight Manager home page of the active and standby clusters, choose O&M> Log > Download.
- Select HBase in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager home page of the active and standby clusters, choose O&M> Log > Download.
- Select HBase in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-19012.html b/docs/mrs/umn/ALM-19012.html index e3c45745..d0ce9e43 100644 --- a/docs/mrs/umn/ALM-19012.html +++ b/docs/mrs/umn/ALM-19012.html @@ -3,9 +3,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-19012 HBase System Table Directory or File Lost
Description
The system checks whether HBase directories and files exist on the HDFS every 120 seconds. This alarm is generated when the system detects that the files or directories do not exist. This alarm is cleared when the files or directories are restored.
The HBase directories and files are as follows:
-+
- Directory of the namespace hbase on the HDFS
- hbase.version file
- Directory of the table hbase:meta on the HDFS, .tableinfo file, and .regioninfo file
- Directory of the table hbase:namespace on the HDFS, .tableinfo file, and .regioninfo file
- Directory of the table hbase:hindex on the HDFS, .tableinfo file, and .regioninfo file
- Directory of the hbase:acl table on the HDFS, .tableinfo, and .regioninfo file (This table does not exist in the common mode cluster by default.)
-If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the HBase1 service is unavailable, ServiceName=HBase1 is displayed in Location, and the operation object in the procedure needs to be changed from HBase to HBase1.
-
- Directory of the namespace hbase on the HDFS
- hbase.version file
- Directory of the table hbase:meta on the HDFS, .tableinfo file, and .regioninfo file
- Directory of the table hbase:namespace on the HDFS, .tableinfo file, and .regioninfo file
- Directory of the table hbase:hindex on the HDFS, .tableinfo file, and .regioninfo file
- Directory of the hbase:acl table on the HDFS, .tableinfo, and .regioninfo file (This table does not exist in the common mode cluster by default.)
Attribute
Alarm ID
@@ -67,7 +65,7 @@- On the FusionInsight Manager home page, choose O&M > Backup and Restoration > Backup Management. Check whether there are success records of the backup task named default or other HBase metadata backup tasks that have been successfully executed.
- Use the latest backup metadata to restore the metadata of the HBase service.
Collect fault information.
-+
- On the FusionInsight Manager page of the active and standby clusters, choose O&M > Log > Download.
- In the Service area, select faulty HBase services in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager page of the active and standby clusters, choose O&M > Log > Download.
- In the Service area, select faulty HBase services in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-19013.html b/docs/mrs/umn/ALM-19013.html index 30f87fe4..31e7d6be 100644 --- a/docs/mrs/umn/ALM-19013.html +++ b/docs/mrs/umn/ALM-19013.html @@ -1,9 +1,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-19013 Duration of Regions in transaction State Exceeds the Threshold
-Description
The system checks the number of regions in transaction state on HBase every 300 seconds. This alarm is generated when the system detects that the duration of regions in transaction state exceeds the threshold for two consecutive times. This alarm is cleared when all timeout regions are restored.+-If the multi-instance function is enabled in the cluster and multiple HBase service instances are installed, you need to determine the HBase service instance where the alarm is generated based on the value of ServiceName in Location. For example, if the HBase1 service is unavailable, ServiceName=HBase1 is displayed in Location, and the operation object in the procedure needs to be changed from HBase to HBase1.
-Description
The system checks the number of regions in transaction state on HBase every 300 seconds. This alarm is generated when the system detects that the duration of regions in transaction state exceeds the threshold for two consecutive times. This alarm is cleared when all timeout regions are restored.
Attribute
Alarm ID
@@ -75,7 +73,7 @@- Repeat 4. Run the hbase hbck command on the client and check whether the error message "No table descriptor file under hdfs://hacluster/hbase/data/default/table" is displayed.
- If yes, go to 7.
- If no, no further action is required.
Collect fault information.
-+
- On the FusionInsight Manager page of the active and standby clusters, choose O&M > Log > Download.
- In the Service area, select faulty HBase services in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager page of the active and standby clusters, choose O&M > Log > Download.
- In the Service area, select faulty HBase services in the required cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-19014.html b/docs/mrs/umn/ALM-19014.html index 7d01def3..967bdf43 100644 --- a/docs/mrs/umn/ALM-19014.html +++ b/docs/mrs/umn/ALM-19014.html @@ -3,8 +3,6 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
ALM-19014 Capacity Quota Usage on ZooKeeper Exceeds the Threshold Severely
Description
The system checks the ZNode usage of the HBase service every 120 seconds. This alarm is generated when the ZNode capacity usage of the HBase service exceeds the critical alarm threshold (90% by default).
This alarm is cleared when the ZNode capacity usage is less than the critical alarm threshold.
-If the multi-instance function has been enabled in the cluster and multiple HBase services have been installed, determine the HBase service for which the alarm is generated based on the value of ServiceName in Location. For example, if the value of ServiceName is HBase-1, change the operation object in the procedure from HBase to HBase-1.
-Attribute
Alarm ID
@@ -73,7 +71,7 @@If the cluster uses the security mode, run the following command to perform security authentication:
kinit hbase
Enter the password as prompted (obtain the password from the MRS cluster administrator).
-- Run the hbase zkcli command to log in to the ZooKeeper client and run the listquota /hbase command to check the ZNode capacity quota of the HBase service. The ZNode root directory in the command is specified by the zookeeper.znode.parent parameter of the HBase service. The marked area in the following figure shows the capacity configuration of the root ZNode of the HBase service.
+
- Run the hbase zkcli command to log in to the ZooKeeper client and run the listquota /hbase command to check the ZNode capacity quota of the HBase service. The ZNode root directory in the command is specified by the zookeeper.znode.parent parameter of the HBase service. The marked area in the following figure shows the capacity configuration of the root ZNode of the HBase service.
- Run the getusage /hbase/splitWAL command to check the capacity usage of the ZNode. Check whether the ratio of Data size to the ZNode capacity quota is close to the alarm threshold.
- On FusionInsight Manager, choose O&M > Alarm > Alarms. Check whether the alarm whose ID is 12007, 19000, or 19013 and the ServiceName in Location is the current HBase service exists.
- Run the getusage /hbase/replication command to check the capacity usage of the ZNode. Check whether the ratio of Data size to the ZNode capacity quota is close to the alarm threshold. @@ -81,7 +79,7 @@
- Check whether the alarm is cleared five minutes later.
- If yes, no further action is required.
- If no, go to 9.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-19015.html b/docs/mrs/umn/ALM-19015.html index e34456df..c7508e8f 100644 --- a/docs/mrs/umn/ALM-19015.html +++ b/docs/mrs/umn/ALM-19015.html @@ -3,8 +3,6 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-19015 Quantity Quota Usage on ZooKeeper Exceeds the Threshold
Description
The system checks the ZNode usage of the HBase service every 120 seconds. This alarm is generated when the system detects that the ZNode quantity usage of the HBase service exceeds the alarm threshold (75% by default).
This alarm is cleared when the ZNode quantity usage is less than the alarm threshold.
-If the multi-instance function has been enabled in the cluster and multiple HBase services have been installed, determine the HBase service for which the alarm is generated based on the value of ServiceName in Location. For example, if the value of ServiceName is HBase-1, change the operation object in the procedure from HBase to HBase-1.
-Attribute
Alarm ID
@@ -73,7 +71,7 @@If the cluster uses the security mode, run the following command to perform security authentication:
kinit hbase
Enter the password as prompted (obtain the password from the MRS cluster administrator).
-- Run the hbase zkcli command to log in to the ZooKeeper client and run the listquota /hbase command to check the ZNode quantity quota of the HBase service. The ZNode root directory in the command is specified by the zookeeper.znode.parent parameter of the HBase service. The marked area in the following figure shows the quantity quota configuration of the root ZNode of the HBase service.
+
- Run the hbase zkcli command to log in to the ZooKeeper client and run the listquota /hbase command to check the ZNode quantity quota of the HBase service. The ZNode root directory in the command is specified by the zookeeper.znode.parent parameter of the HBase service. The marked area in the following figure shows the quantity quota configuration of the root ZNode of the HBase service.
- Run the getusage /hbase/splitWAL command to check the ZNode quantity usage and check whether the ratio of Node count in the command output to the ZNode quantity quota is close to the alarm threshold.
- On FusionInsight Manager, choose O&M > Alarm > Alarms. Check whether the alarm whose ID is 12007, 19000, or 19013 and the ServiceName in Location is the current HBase service exists.
- Run the getusage /hbase/replication command to check the ZNode quantity usage and check whether the ratio of Node count in the command output to the ZNode quantity quota is close to the alarm threshold. @@ -81,7 +79,7 @@
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-19016.html b/docs/mrs/umn/ALM-19016.html index 133b4184..0116eaf4 100644 --- a/docs/mrs/umn/ALM-19016.html +++ b/docs/mrs/umn/ALM-19016.html @@ -3,8 +3,6 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-19016 Quantity Quota Usage on ZooKeeper Exceeds the Threshold Severely
Description
The system checks the ZNode usage of the HBase service every 120 seconds. This alarm is generated when the znode usage of the HBase service exceeds the critical alarm threshold (90% by default).
This alarm is cleared when the quantity usage of the ZNode is less than the critical alarm threshold.
-If the multi-instance function has been enabled in the cluster and multiple HBase services have been installed, determine the HBase service for which the alarm is generated based on the value of ServiceName in Location. For example, if the value of ServiceName is HBase-1, change the operation object in the procedure from HBase to HBase-1.
-Attribute
Alarm ID
@@ -73,7 +71,7 @@If the cluster uses the security mode, run the following command to perform security authentication:
kinit hbase
Enter the password as prompted (obtain the password from the MRS cluster administrator).
-- Run the hbase zkcli command to log in to the ZooKeeper client and run the listquota /hbase command to check the ZNode quantity quota of the HBase service. The ZNode root directory in the command is specified by the zookeeper.znode.parent parameter of the HBase service. The marked area in the following figure shows the quantity configuration of the root ZNode of the HBase service.
+
- Run the hbase zkcli command to log in to the ZooKeeper client and run the listquota /hbase command to check the ZNode quantity quota of the HBase service. The ZNode root directory in the command is specified by the zookeeper.znode.parent parameter of the HBase service. The marked area in the following figure shows the quantity configuration of the root ZNode of the HBase service.
- Run the getusage /hbase/splitWAL command to check the ZNode usage and check whether the ratio of Node count in the command output to the znode quantity quota is close to the alarm threshold.
- On FusionInsight Manager, choose O&M > Alarm > Alarms. Check whether the alarm whose ID is 12007, 19000, or 19013 and the ServiceName in Location is the current HBase service exists.
- Run the getusage /hbase/replication command to check the ZNode usage and check whether the ratio of Node count in the command output to the ZNode quantity quota is close to the alarm threshold. @@ -81,7 +79,7 @@
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-19017.html b/docs/mrs/umn/ALM-19017.html index f68db87a..c83290ab 100644 --- a/docs/mrs/umn/ALM-19017.html +++ b/docs/mrs/umn/ALM-19017.html @@ -3,8 +3,6 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-19017 Capacity Quota Usage on ZooKeeper Exceeds the Threshold
Description
The system checks the ZNode usage of the HBase service every 120 seconds. This alarm is generated when the system detects that the ZNodes capacity usage of the HBase service exceeds the alarm threshold (75% by default).
This alarm is cleared when the capacity usage of the ZNode capacity is less than the threshold.
-If the multi-instance function has been enabled in the cluster and multiple HBase services have been installed, determine the HBase service for which the alarm is generated based on the value of ServiceName in Location. For example, if the value of ServiceName is HBase-1, change the operation object in the procedure from HBase to HBase-1.
-Attribute
Alarm ID
@@ -73,7 +71,7 @@If the cluster uses the security mode, run the following command to perform security authentication:
kinit hbase
Enter the password as prompted (obtain the password from the MRS cluster administrator).
-- Run the hbase zkcli command to log in to the ZooKeeper client and run the listquota /hbase command to check the ZNode quantity quota of the HBase service. The ZNode root directory in the command is specified by the zookeeper.znode.parent parameter of the HBase service. The marked area in the following figure shows the quantity configuration of the root ZNode of the HBase service.
+
- Run the hbase zkcli command to log in to the ZooKeeper client and run the listquota /hbase command to check the ZNode quantity quota of the HBase service. The ZNode root directory in the command is specified by the zookeeper.znode.parent parameter of the HBase service. The marked area in the following figure shows the quantity configuration of the root ZNode of the HBase service.
- Run the getusage /hbase/splitWAL command to check the capacity usage of the ZNode. Check whether the ratio of Data size to the ZNode capacity quota is close to the alarm threshold.
- On FusionInsight Manager, check whether the alarm whose ID is 12007, 19000, or 19013 and ServiceName in Location is the current HBase service exists.
- Run the getusage /hbase/replication command to check the capacity usage of the ZNode. Check whether the ratio of Data size to the ZNode capacity quota is close to the alarm threshold. @@ -81,7 +79,7 @@
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-19018.html b/docs/mrs/umn/ALM-19018.html index c1443f72..85c86b5e 100644 --- a/docs/mrs/umn/ALM-19018.html +++ b/docs/mrs/umn/ALM-19018.html @@ -2,8 +2,6 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-19018 HBase Compaction Queue Size Exceeds the Threshold
Description
The system checks the HBase compaction queue size every 300 seconds. This alarm is generated when the compaction queue size exceeds the alarm threshold (100 by default). This alarm is cleared when the compaction queue size is less than the threshold.
-If the multi-instance function has been enabled in the cluster and multiple HBase services have been installed, determine the HBase service for which the alarm is generated based on the value of ServiceName in Location. For example, if the value of ServiceName is HBase-1, change the operation object in the procedure from HBase to HBase-1.
-Attribute
Alarm ID
@@ -62,10 +60,10 @@Procedure
Check whether related parameters are properly configured.
- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. On the page that is displayed, check whether the alarm whose Alarm ID is 19008 or 19011 exists. -
- On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > HBase. On the page that is disaplyed, click the Configuration tab then the All Configurations sub-tab, search for hbase.hstore.compaction.min, hbase.hstore.compaction.max, hbase.regionserver.thread.compaction.small, and hbase.regionserver.thread.compaction.throttle, and set them to larger values.
- Check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 4.
- On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > HBase. On the page that is displayed, click the Configuration tab then the All Configurations sub-tab, search for hbase.hstore.compaction.min, hbase.hstore.compaction.max, hbase.regionserver.thread.compaction.small, and hbase.regionserver.thread.compaction.throttle, and set them to larger values.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-19019.html b/docs/mrs/umn/ALM-19019.html index ae99c9bd..0a9dae73 100644 --- a/docs/mrs/umn/ALM-19019.html +++ b/docs/mrs/umn/ALM-19019.html @@ -87,7 +87,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 15.
Collect the fault information.
-+
- On FusionInsight Manager of the standby cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager of the standby cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-19020.html b/docs/mrs/umn/ALM-19020.html index f3393b23..a4798b1f 100644 --- a/docs/mrs/umn/ALM-19020.html +++ b/docs/mrs/umn/ALM-19020.html @@ -87,7 +87,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 15.
Collect the fault information.
-+
- On FusionInsight Manager of the standby cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager of the standby cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-19021.html b/docs/mrs/umn/ALM-19021.html new file mode 100644 index 00000000..6ba5775b --- /dev/null +++ b/docs/mrs/umn/ALM-19021.html @@ -0,0 +1,112 @@ + + +Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-19021 Handler Usage of RegionServer Exceeds the Threshold
+++Description
The system checks the RegionServer handler usage of each HBase service instance every 30 seconds. This alarm is generated when the handler usage of a RegionServer exceeds the threshold (90% for five consecutive times by default). This alarm is cleared if the handler usage is lower than or equal to the threshold.
++This section applies to MRS 3.2.0 or later.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Auto Clear
++ + + 19021
++ Major
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + HostName
++ Specifies the host for which the alarm is generated.
++ + + Trigger Condition
++ Specifies the threshold for triggering the alarm.
++Impact on the System
RegionServers or HBase cannot provide services properly.
++Possible Causes
+
- The value of a handler is too small.
- Hotspotting occurs.
+Procedure
View alarm location information.
++
- Log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Alarm > Alarms, locate the row that contains the alarm whose Alarm ID is 19021, and view the service instance and host name in Location.
Check the handler configuration.
++
- Choose Cluster > Services > HBase and click the Configurations tab. In the upper right corner of the page, search for hbase.regionserver.handler.count and check whether its value is too small. The default value is 200. +
- Change the value of this parameter to a greater value and save the configuration. Choose Cluster > Services > HBase, click the Instance tab, select the affected RegionServer instances, and choose More > Instance Rolling Restart. In the displayed dialog box, enter the username and password. In the Instance Rolling Restart dialog box, click OK and wait until the rolling restart is complete.
- After the configuration takes effect, check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 5.
Check whether cluster hotspotting occurs.
++
- On FusionInsight Manager, choose Cluster > Services > HBase and click HMaster(Active) following HMaster WebUI to go to the web UI of the HBase instance. In the Region Servers area of the Home page, click Requests and check whether the requests in the Filtered Read Request Count and Write Request Count columns are evenly distributed.
+ +
- Check whether regions are evenly distributed.
On FusionInsight Manager, choose Cluster > Services > HBase and click HMaster(Active) following HMaster WebUI to go to the web UI of the HBase instance. In the Region Servers area of the Home page, click Base Stats and check whether the regions in the Num.Regions column are evenly distributed.
++ +
- Log in to the faulty RegionServer node as user omm.
- Run the following commands to go to the client installation directory and set the environment variable:
cd Client installation directory
+source bigdata_env
+If the cluster uses the security mode, run the following command to perform security authentication:
+kinit hbase
+Enter the password as prompted (obtain the password from the MRS cluster administrator).
+- Run the following commands to check whether the load balancing function is enabled. If the command output is true, the load balancing function is enabled.
hbase shell
+balancer_enabled
+hbase:004:0> balancer_enabled +true +Took 0.0165 seconds +=> true+ +- Run the following commands in HBase Shell to enable the load balancing function and check whether the function is enabled.
balance_switch true
+balancer_enabled
++You are advised to enable and manually trigger the load balancing function during off-peak hours.
+- Run the following command to manually trigger' the load balancing function:
balancer
+- After the load balancing is complete, log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Alarm > Alarms and check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 13.
Collect the fault information.
++
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
+Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
++Related Information
None
+++ diff --git a/docs/mrs/umn/ALM-20002.html b/docs/mrs/umn/ALM-20002.html index cdd7e3ee..52d97240 100644 --- a/docs/mrs/umn/ALM-20002.html +++ b/docs/mrs/umn/ALM-20002.html @@ -75,7 +75,7 @@++Parent topic: Alarm Reference (Applicable to MRS 3.x)+Collect fault information.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service drop-down list:
-
- Hue
- Controller
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- On the FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Hue.
- Choose More > Restart Service, and click OK.
- Check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 19.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- On the FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Hue.
- Choose More > Restart Service, and click OK.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 19.
- Contact the O&M personnel and send the collected logs.
Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
diff --git a/docs/mrs/umn/ALM-23001.html b/docs/mrs/umn/ALM-23001.html new file mode 100644 index 00000000..7715a6b1 --- /dev/null +++ b/docs/mrs/umn/ALM-23001.html @@ -0,0 +1,106 @@ + + +ALM-23001 Loader Service Unavailable
+++Description
The system checks the Loader service availability every 60 seconds. This alarm is generated when the system detects that the Loader service is unavailable. This alarm is cleared when the Loader service is available.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Automatically Cleared
++ + + 23001
++ Critical
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + + HostName
++ Specifies the host for which the alarm is generated.
++Impact on the System
When the Loader service is unavailable, the data loading, import, and conversion functions are unavailable.
++Possible Causes
+
- The internal service on which the Loader service depends is abnormal.
+
- The ZooKeeper service is abnormal.
- The HDFS service is abnormal.
- The DBService service is abnormal.
- The Yarn service is abnormal.
- The Mapreduce service is abnormal.
- Environment fault: The network is abnormal, which the Loader service cannot communicate with the depended internal services and cannot provide services.
- Software fault: The Loader service cannot run properly.
+Procedure
Check the ZooKeeper service status.
++
- On the FusionInsight Manager home page, choose Cluster > Name of the desired cluster > Services > ZooKeeper to check whether the ZooKeeper running status is Normal. +
- Choose More > Restart Service to restart the ZooKeeper service. In the alarm list, check whether LoaderService Unavailable is cleared.
+
- If yes, no further action is required.
- If no, go to 3.
- On the FusionInsight Manager, check whether the alarm list contains Process Fault. +
- In the Location area of Process Fault, check whether ServiceName is ZooKeeper. +
- Rectify the fault by following the steps provided in ALM-12007 Process Fault.
- In the alarm list, check whether Loader Service Unavailable is cleared.
+
- If yes, no further action is required.
- If no, go to 7.
Check the HDFS service status.
++
- On the FusionInsight Manager, check whether the alarm list contains HDFS Service Unavailable. +
- Rectify the fault by following the steps provided in ALM-14000 HDFS Service Unavailable.
- In the alarm list, check whether Loader Service Unavailable is cleared.
+
- If yes, no further action is required.
- If no, go to 10.
Check the DBService status.
++
- On the FusionInsight Manager home page, choose Cluster > Name of the desired cluster > Services > DBService to check whether the DBService running status is Normal. +
- Choose More > Restart Service to restart the DBService service. In the alarm list, check whether LoaderService Unavailable is cleared.
+
- If yes, no further action is required.
- If no, go to 12.
Check the Mapreduce status.
++
- On the FusionInsight Manager home page, choose Cluster > Name of the desired cluster > Services > Mapreduce to check whether the Mapreduce running status is Normal. +
- Choose More > Restart Service to restart the Mapreduce service. In the alarm list, check whether LoaderService Unavailable is cleared.
+
- If yes, no further action is required.
- If no, go to 16.
Check the Yarn status.
++
- On the FusionInsight Manager home page, choose Cluster > Name of the desired cluster > Services > Yarn to check whether the Yarn running status is Normal. +
- Choose More > Restart Service to restart the Yarn service. In the alarm list, check whether LoaderService Unavailable is cleared.
+
- If yes, no further action is required.
- If no, go to 16.
- On the FusionInsight Manager, check whether the alarm list contains Yarn Service Unavailable. +
- Rectify the fault by following the steps provided in ALM-18000 Yarn Service Unavailable.
- In the alarm list, check whether Loader Service Unavailable is cleared.
+
- If yes, no further action is required.
- If no, go to 19.
Check the network connection between Loader and dependent components.
+
- On the FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Loader.
- Click Instance and the LoaderServer instance list is displayed.
- Record the Management IP Address in the row of LoaderServer(Active).
- Log in to the host where the active LoaderServer runs as omm user using the IP address obtained in 21.
+
- Run the ping command to check whether communication between the host that runs the active LoaderServer and the hosts that run the dependent components. (The dependent components include ZooKeeper, DBService, HDFS, Mapreduce and Yarn. Obtain the IP addresses of the hosts that run these services in the same way as that for obtaining the IP address of the active LoaderServer.) +
- Contact the administrator to restore the network.
- In the alarm list, check whether Loader Service Unavailable is cleared.
+
- If yes, no further action is required.
- If no, go to 26.
Collect fault information.
+
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service drop-down list:
+
- ZooKeeper
- HDFS
- DBService
- Yarn
- Mapreduce
- Loader
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- On the FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Loader.
- Choose More > Restart Service, and click OK.
+
- Check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 32.
- Contact the O&M personnel and send the collected logs.
+Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
++Related Information
None
+++ diff --git a/docs/mrs/umn/ALM-23003.html b/docs/mrs/umn/ALM-23003.html new file mode 100644 index 00000000..2b6a8f38 --- /dev/null +++ b/docs/mrs/umn/ALM-23003.html @@ -0,0 +1,104 @@ + + +++Parent topic: Alarm Reference (Applicable to MRS 3.x)+ALM-23003 Loader Task Execution Failure
+++Description
This alarm is generated immediately when the system detects that the Loader job fails. This alarm is cleared when the failed job is manually handled by a user. This alarm must be manually cleared.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Automatically Cleared
++ + + 23003
++ Minor
++ No
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + HostName
++ Specifies the host for which the alarm is generated.
++ + JobID
++ Specifies the ID of failed Loader job.
++ + JobName
++ Specifies the failed Loader job.
++ + UserName
++ Specifies the name of the user who submits the Loader job.
++ + + Details
++ Supplementary information for which the alarm is generated.
++Impact on the System
An exception occurs during the execution of the submitted Loader job, and the execution fails. No execution result can be obtained. Execute the job again after rectifying the fault.
++Possible Causes
+
- Task parameters are incorrectly configured.
- Exceptions occur when Yarn is executing a job.
+Procedure
Check whether task parameters are incorrectly configured.
++
- On FusionInsight Manager, choose O&M > Alarm > Alarms and click the alarm drop-down list from the alarm list, obtain the Alarm Cause.
- If the alarm cause is "Failure to submit job", view error details in Additional Information, and go to the Loader WebUI to view the execution history of the job.
+By default, the admin user does not have the permissions to manage other components. If the page cannot be opened or the displayed content is incomplete when you access the native UI of a component due to insufficient permissions, you can manually create a user with the permissions to manage that component.
+- Submit the task again.
- Check whether the task executed successfully. +
Check whether exceptions occur when Yarn is executing a job.
++
- On FusionInsight Manager, click the alarm drop-down list from the alarm list, obtain the Alarm Cause.
- Check whether the Yarn activity is executed properly in the Alarm Cause. If the alarm cause is "Yarn execution failed", the Yarn activity is abnormal. +
- Submit the task again.
- Please check whether the task executed successfully. +
- In the alarm list, click Clear from Operation to manually clear the alarm. No further action is required.
Collect fault information.
++
- On FusionInsight Manager, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service drop-down list:
+
- DBService
- HDFS
- Loader
- Mapreduce
- Yarn
- ZooKeeper
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
+Alarm Clearing
After the fault is rectified, the system does not automatically clear this alarm, and you need to manually clear the alarm.
++Related Information
None
+++ diff --git a/docs/mrs/umn/ALM-23004.html b/docs/mrs/umn/ALM-23004.html new file mode 100644 index 00000000..040acafa --- /dev/null +++ b/docs/mrs/umn/ALM-23004.html @@ -0,0 +1,85 @@ + + +++Parent topic: Alarm Reference (Applicable to MRS 3.x)+ALM-23004 Loader Heap Memory Usage Exceeds the Threshold
+++Description
The system checks the heap memory usage of the Loader service every 60 seconds. The alarm is generated when the heap memory usage of a Loader instance exceeds the threshold (95% of the maximum memory) for 10 consecutive times. The alarm is cleared when the heap memory usage is less than the threshold.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Automatically Cleared
++ + + 23004
++ Major
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + HostName
++ Specifies the host for which the alarm is generated.
++ + + Trigger Condition
++ Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.
++Impact on the System
The heap memory overflow may cause a service breakdown.
++Possible Causes
The heap memory of the Loader instance is overused or the heap memory is inappropriately allocated.
++Procedure
Check heap memory usage.
++
- On the FusionInsight Manager portal, choose O&M > Alarm > Alarms > Loader Heap Memory Usage Exceeds the Threshold > Location. Check the host name of the instance involved in this alarm.
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Loader > Instance. Click the instance for which the alarm is generated to go to the page for the instance. Click the drop-down menu in the chart area and choose Customize > Memory > Loader Heap Memory Resource Percentage. Click OK.
- Check whether the used heap memory of Loader reaches the threshold (the default value is 95% of the maximum heap memory) specified for Loader. +
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Loader > Configurations. Click All Configurations. Increase the value of -Xmx in GC_OPTS as required, and click Save. Click OK.
++
- If this alarm is generated, the heap memory configured for the current Loader instance is insufficient for data transmission. You are advised to open the instance monitoring page, display the Loader heap memory resource status monitoring chart, and observe the change trend of the heap memory used by Loader in the monitoring chart. Then change the value of -Xmx to twice the current heap memory usage or to another value to meet site requirements.
- When setting the heap memory, you can set -Xms and -Xmx to similar values to avoid performance deterioration caused by heap size adjustment after each GC.
- Ensure that the sum of -Xmx and XX:MaxPermSize is not greater than the physical memory of the node server.
- Restart the affected services or instances and check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
++
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Loader in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
+Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
++Related Information
None
+++ diff --git a/docs/mrs/umn/ALM-23005.html b/docs/mrs/umn/ALM-23005.html new file mode 100644 index 00000000..4dd2e27d --- /dev/null +++ b/docs/mrs/umn/ALM-23005.html @@ -0,0 +1,85 @@ + + +++Parent topic: Alarm Reference (Applicable to MRS 3.x)+ALM-23005 Loader Non-Heap Memory Usage Exceeds the Threshold
+++Description
The system checks the non-heap memory usage of the Loader service every 30 seconds. The alarm is generated when the non-heap memory usage of a Loader instance exceeds the threshold (80% of the maximum memory) for 5 consecutive times. The alarm is cleared when the non-heap memory usage is less than the threshold.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Automatically Cleared
++ + + 23005
++ Major
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + HostName
++ Specifies the host for which the alarm is generated.
++ + + Trigger Condition
++ Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.
++Impact on the System
The non-heap memory overflow may cause a service breakdown.
++Possible Causes
The non-heap memory of the Loader instance is overused or the non-heap memory is inappropriately allocated.
++Procedure
Check non-heap memory usage.
++
- On the FusionInsight Manager portal, choose O&M > Alarm > Alarms > Loader Non-Heap Memory Usage Exceeds the Threshold > Location. Check the host name of the instance involved in this alarm.
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Loader > Instance. Click the instance for which the alarm is generated to go to the page for the instance. Click the drop-down menu in the chart area and choose Customize > Memory > Loader Non Heap Memory Resource Percentage. Click OK.
- Check whether the used non-heap memory of Loader reaches the threshold (the default value is 80% of the maximum non-heap memory) specified for Loader. +
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Loader > Configurations. Click All Configurations Search LOADER_GC_OPTS in the search box. If the -XX: MaxPermSize parameter is not configured, set the initial value to -XX: MaxPermSize=256M for the first time. (If the alarm persists after the first adjustment, perform the second adjustment by referring to the following note.) And click Save. Click OK.
+If this alarm is generated, the non-heap memory configured for the current Loader instance is insufficient for the service scenario. You are advised to open the instance monitoring page, open the Loader non-heap memory resource status monitoring chart, and observe the change trend of the non-heap memory used by Loader in the monitoring chart. Then change the value of -XX:MaxPermSize to twice the current non-heap memory usage or to another value to meet site requirements.
+- Restart the affected services or instances and check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
++
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Loader in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
+Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
++Related Information
None
+++ diff --git a/docs/mrs/umn/ALM-23006.html b/docs/mrs/umn/ALM-23006.html new file mode 100644 index 00000000..dfc2cfb0 --- /dev/null +++ b/docs/mrs/umn/ALM-23006.html @@ -0,0 +1,85 @@ + + +++Parent topic: Alarm Reference (Applicable to MRS 3.x)+ALM-23006 Loader Direct Memory Usage Exceeds the Threshold
+++Description
The system checks the direct memory usage of the Loader service every 30 seconds. The alarm is generated when the direct memory usage of a Loader instance exceeds the threshold (80% of the maximum memory) for 5 consecutive times. The alarm is cleared when the direct memory usage of Loader is less than or equal to the threshold.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Automatically Cleared
++ + + 23006
++ Major
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + HostName
++ Specifies the host for which the alarm is generated.
++ + + Trigger Condition
++ Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.
++Impact on the System
The direct memory overflow may cause a service breakdown.
++Possible Causes
The direct memory of the Loader instance is overused or the direct memory is inappropriately allocated.
++Procedure
Check direct memory usage.
++
- On the FusionInsight Manager portal, choose O&M > Alarm > Alarms > Loader Direct Memory Usage Exceeds the Threshold > Location. Check the host name of the instance involved in this alarm.
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Loader > Instance. Click the instance for which the alarm is generated to go to the page for the instance. Click the drop-down menu in the chart area and choose Customize > Memory > Loader Memory Usage Statistics. Click OK.
- Check whether the used direct memory of Loader reaches the threshold (the default value is 80% of the maximum direct memory) specified for Loader. +
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Loader > Configurations. Click All Configurations. Search LOADER_GC_OPTS in the search box. Increase the value of -XX:MaxDirectMemorySize as required, and click Save. Click OK.
+If this alarm is generated, the direct memory configured for the current Loader instance is insufficient for the service scenario. You are advised to open the instance monitoring page, display the Loader direct memory resource status monitoring chart, and observe the change trend of the direct memory used by Loader in the monitoring chart. Then change the value of -XX:MaxDirectMemorySize to twice the current direct memory usage or to another value to meet site requirements.
+- Restart the affected services or instances and check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
++
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Loader in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
+Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
++Related Information
None
+++ diff --git a/docs/mrs/umn/ALM-23007.html b/docs/mrs/umn/ALM-23007.html new file mode 100644 index 00000000..04680f4e --- /dev/null +++ b/docs/mrs/umn/ALM-23007.html @@ -0,0 +1,85 @@ + + +++Parent topic: Alarm Reference (Applicable to MRS 3.x)+ALM-23007 Garbage Collection (GC) Time of the Loader Process Exceeds the Threshold
+++Description
The system checks GC time of the Loader process every 60 seconds. The alarm is generated when GC time of the Loader process exceeds the threshold (default value: 12 seconds) for 5 consecutive times. The alarm is cleared when GC time is less than the threshold.
++Attribute
+++
+ + + Alarm ID
++ Alarm Severity
++ Automatically Cleared
++ + + 23007
++ Major
++ Yes
++Parameters
+++
+ + + Name
++ Meaning
++ + Source
++ Specifies the cluster for which the alarm is generated.
++ + ServiceName
++ Specifies the service for which the alarm is generated.
++ + RoleName
++ Specifies the role for which the alarm is generated.
++ + HostName
++ Specifies the host for which the alarm is generated.
++ + + Trigger Condition
++ Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.
++Impact on the System
Loader service response is slow.
++Possible Causes
The heap memory of the Loader instance is overused or the heap memory is inappropriately allocated.
++Procedure
Check GC time.
++
- On the FusionInsight Manager portal, choose O&M > Alarm > Alarms > Garbage Collection (GC) Time of the Loader Process Exceeds the Threshold > Location. Check the host name of the instance involved in this alarm.
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Loader > Instance. Click the instance for which the alarm is generated to go to the page for the instance. Click the drop-down menu in the chart area and choose Customize > GC > Garbage Collection (GC) Time of Loader. Click OK.
- Check whether GC time of the Loader process every second exceeds the threshold (default value: 12 seconds). +
- On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Loader > Configurations. Click All Configurations. Search LOADER_GC_OPTS in the search box. Increase the value of -Xmx as required, and click Save. Click OK.
+If this alarm is generated, the heap memory configured for the current Loader instance cannot meet the heap memory required for data transmission. You are advised to handle the problem by referring to 4 in section ALM-23004 Loader Heap Memory Usage Exceeds the Threshold.
+- Restart the affected services or instances and check whether the alarm is cleared.
+
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
++
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Loader in the required cluster from the Service drop-down list.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
+Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
++Related Information
None
+++ diff --git a/docs/mrs/umn/ALM-24000.html b/docs/mrs/umn/ALM-24000.html index 07257030..81bfa367 100644 --- a/docs/mrs/umn/ALM-24000.html +++ b/docs/mrs/umn/ALM-24000.html @@ -63,7 +63,7 @@++Parent topic: Alarm Reference (Applicable to MRS 3.x)+- In the alarm list, check whether alarm "Flume Service Unavailable" is cleared.
- If yes, no further action is required.
- If no, go to 3.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24001.html b/docs/mrs/umn/ALM-24001.html index 38cbbf75..cda4714e 100644 --- a/docs/mrs/umn/ALM-24001.html +++ b/docs/mrs/umn/ALM-24001.html @@ -78,9 +78,9 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
Check the Flume agent configuration.
- Run the cat properties.properties | grep spooldir and cat properties.properties | grep TAILDIR commands to check whether the Flume source type is spoolDir or tailDir. If any command output is displayed, the Flume source type is spoolDir or tailDir.
- Check whether the data monitoring directory exists.
- Specify a correct data monitoring directory.
- Check whether the Flume agent user has the read, write, and execute permissions on the monitoring directory specified in 13.
- Run the ll ketab path command to check whether the keytab authentication path specified by the *.kerberosKeytab parameter in the configuration file exists.
- Change the value of kerberosKeytab in 18 to the custom keytab path and go to 21.
- Perform 18 to check whether the Flume agent running user has the permission to access the keytab authentication file. If the keytab path is returned, the user has the permission. Otherwise, the user does not have the permission.
- Run the chmod 755 ketab file command to grant the read permission on the keytab file specified in 19, and restart the Flume process.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 23.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24003.html b/docs/mrs/umn/ALM-24003.html index 3aa904fe..97aa4826 100644 --- a/docs/mrs/umn/ALM-24003.html +++ b/docs/mrs/umn/ALM-24003.html @@ -72,7 +72,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Collect logs in the /var/log/Bigdata/flume-client directory on the Flume client using a transmission tool.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Collect logs in the /var/log/Bigdata/flume-client directory on the Flume client using a transmission tool.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24004.html b/docs/mrs/umn/ALM-24004.html index a213a05b..df38cd1d 100644 --- a/docs/mrs/umn/ALM-24004.html +++ b/docs/mrs/umn/ALM-24004.html @@ -85,7 +85,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Contact the network administrator to restore the network.
- In the alarm list, check whether the alarm is cleared after a period.
- If yes, no further action is required.
- If no, go to 11.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24005.html b/docs/mrs/umn/ALM-24005.html index 18bf671d..cf088f96 100644 --- a/docs/mrs/umn/ALM-24005.html +++ b/docs/mrs/umn/ALM-24005.html @@ -85,7 +85,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Contact the network administrator to restore the network.
- In the alarm list, check whether the alarm is cleared after a period.
- If yes, no further action is required.
- If no, go to 13.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24006.html b/docs/mrs/umn/ALM-24006.html index a80e4e15..9477b823 100644 --- a/docs/mrs/umn/ALM-24006.html +++ b/docs/mrs/umn/ALM-24006.html @@ -70,7 +70,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24007.html b/docs/mrs/umn/ALM-24007.html index 5c65e3e4..32a27eff 100644 --- a/docs/mrs/umn/ALM-24007.html +++ b/docs/mrs/umn/ALM-24007.html @@ -70,7 +70,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24008.html b/docs/mrs/umn/ALM-24008.html index bc018e89..2d57ec70 100644 --- a/docs/mrs/umn/ALM-24008.html +++ b/docs/mrs/umn/ALM-24008.html @@ -70,7 +70,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24009.html b/docs/mrs/umn/ALM-24009.html index 4a263b42..77e779fb 100644 --- a/docs/mrs/umn/ALM-24009.html +++ b/docs/mrs/umn/ALM-24009.html @@ -70,7 +70,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24010.html b/docs/mrs/umn/ALM-24010.html index 35ff3157..fd5fe3a4 100644 --- a/docs/mrs/umn/ALM-24010.html +++ b/docs/mrs/umn/ALM-24010.html @@ -1,22 +1,23 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-24010 Flume Certificate File Is Invalid or Damaged
-Description
Flume checks whether the Flume certificate file is valid (whether the certificate exists and whether the certificate format is correct) every hour. This alarm is generated when the certificate file is invalid or damaged. This alarm is automatically cleared when the certificate file becomes valid again.
+This section applies to MRS 3.2.0 or later.
+Description
Flume checks whether the Flume certificate file is valid (whether the certificate exists and whether the certificate format is correct) every hour. This alarm is generated when the certificate file is invalid or damaged. This alarm is automatically cleared when the certificate file becomes valid again.
Attribute
-
Alarm ID
+
- - Alarm ID
Alarm Severity
+- Alarm Severity
Auto Clear
+Auto Clear
24010
+@@ -24,30 +25,30 @@ - 24010
Major
+- Major
Yes
+Yes
Parameters
-
Name
+
- - Name
Meaning
+Meaning
Source
+- - Source
Specifies the cluster for which the alarm is generated.
+Specifies the cluster for which the alarm is generated.
ServiceName
+- - ServiceName
Specifies the service for which the alarm is generated.
+Specifies the service for which the alarm is generated.
RoleName
+- - RoleName
Specifies the role for which the alarm is generated.
+Specifies the role for which the alarm is generated.
HostName
+@@ -70,7 +71,7 @@ - HostName
Specifies the host for which the alarm is generated.
+Specifies the host for which the alarm is generated.
- Check whether this alarm is generated again during periodic system check.
- If yes, go to 9.
- If no, no further action is required.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24011.html b/docs/mrs/umn/ALM-24011.html index 08c52875..9e319521 100644 --- a/docs/mrs/umn/ALM-24011.html +++ b/docs/mrs/umn/ALM-24011.html @@ -1,22 +1,23 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-24011 Flume Certificate File Is About to Expire
-Description
Flume checks whether the Flume certificate file is about to expire every hour. This alarm is generated when the remaining validity period is at most 30 days. This alarm is automatically cleared when the remaining validity period is greater than 30 days.
+This section applies to MRS 3.2.0 or later.
+Description
Flume checks whether the Flume certificate file is about to expire every hour. This alarm is generated when the remaining validity period is at most 30 days. This alarm is automatically cleared when the remaining validity period is greater than 30 days.
Attribute
-
Alarm ID
+
- - Alarm ID
Alarm Severity
+- Alarm Severity
Auto Clear
+Auto Clear
24011
+@@ -24,30 +25,30 @@ - 24011
Major
+- Major
Yes
+Yes
Parameters
-
Name
+
- - Name
Meaning
+Meaning
Source
+- - Source
Specifies the cluster for which the alarm is generated.
+Specifies the cluster for which the alarm is generated.
ServiceName
+- - ServiceName
Specifies the service for which the alarm is generated.
+Specifies the service for which the alarm is generated.
RoleName
+- - RoleName
Specifies the role for which the alarm is generated.
+Specifies the role for which the alarm is generated.
HostName
+@@ -70,7 +71,7 @@ - HostName
Specifies the host for which the alarm is generated.
+Specifies the host for which the alarm is generated.
- Check whether this alarm is generated again during periodic system check.
- If yes, go to 10.
- If no, no further action is required.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24012.html b/docs/mrs/umn/ALM-24012.html index 87700bad..d1ca8067 100644 --- a/docs/mrs/umn/ALM-24012.html +++ b/docs/mrs/umn/ALM-24012.html @@ -1,22 +1,23 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-24012 Flume Certificate File Has Expired
-Description
Flume checks whether its certificate file in the system has expired every hour. This alarm is generated when the server certificate has expired. This alarm is automatically cleared when the certificate file becomes valid again.
+This section applies to MRS 3.2.0 or later.
+Description
Flume checks whether its certificate file in the system has expired every hour. This alarm is generated when the server certificate has expired. This alarm is automatically cleared when the certificate file becomes valid again.
Attribute
-
Alarm ID
+
- - Alarm ID
Alarm Severity
+- Alarm Severity
Auto Clear
+Auto Clear
24012
+@@ -24,30 +25,30 @@ - 24012
Major
+- Major
Yes
+Yes
Parameters
-
Name
+
- - Name
Meaning
+Meaning
Source
+- - Source
Specifies the cluster for which the alarm is generated.
+Specifies the cluster for which the alarm is generated.
ServiceName
+- - ServiceName
Specifies the service for which the alarm is generated.
+Specifies the service for which the alarm is generated.
RoleName
+- - RoleName
Specifies the role for which the alarm is generated.
+Specifies the role for which the alarm is generated.
HostName
+@@ -71,7 +72,7 @@ - HostName
Specifies the host for which the alarm is generated.
+Specifies the host for which the alarm is generated.
- Check whether this alarm is generated again during periodic system check.
- If yes, go to 9.
- If no, no further action is required.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flume for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24013.html b/docs/mrs/umn/ALM-24013.html index a2641324..883fed19 100644 --- a/docs/mrs/umn/ALM-24013.html +++ b/docs/mrs/umn/ALM-24013.html @@ -1,22 +1,23 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-24013 Flume MonitorServer Certificate File Is Invalid or Damaged
-Description
MonitorServer checks whether its certificate file is valid (whether the certificate exists and whether the certificate format is correct) every hour. This alarm is generated when the certificate file is invalid or damaged. This alarm is automatically cleared when the certificate file becomes valid again.
+This section applies to MRS 3.2.0 or later.
+Description
MonitorServer checks whether its certificate file is valid (whether the certificate exists and whether the certificate format is correct) every hour. This alarm is generated when the certificate file is invalid or damaged. This alarm is automatically cleared when the certificate file becomes valid again.
Attribute
-
Alarm ID
+
- - Alarm ID
Alarm Severity
+- Alarm Severity
Auto Clear
+Auto Clear
24013
+@@ -24,30 +25,30 @@ - 24013
Major
+- Major
Yes
+Yes
Parameters
-
Name
+
- - Name
Meaning
+Meaning
Source
+- - Source
Specifies the cluster for which the alarm is generated.
+Specifies the cluster for which the alarm is generated.
ServiceName
+- - ServiceName
Specifies the service for which the alarm is generated.
+Specifies the service for which the alarm is generated.
RoleName
+- - RoleName
Specifies the role for which the alarm is generated.
+Specifies the role for which the alarm is generated.
HostName
+@@ -70,7 +71,7 @@ - HostName
Specifies the host for which the alarm is generated.
+Specifies the host for which the alarm is generated.
- Check whether this alarm is generated again during periodic system check.
- If yes, go to 9.
- If no, no further action is required.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select MonitorServer in the required cluster for Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select MonitorServer in the required cluster for Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24014.html b/docs/mrs/umn/ALM-24014.html index f7a88359..504a0ae0 100644 --- a/docs/mrs/umn/ALM-24014.html +++ b/docs/mrs/umn/ALM-24014.html @@ -1,22 +1,23 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-24014 Flume MonitorServer Certificate Is About to Expire
-Description
MonitorServer checks whether its certificate file is about to expire every hour. This alarm is generated when the remaining validity period is at most 30 days. This alarm is automatically cleared when the remaining validity period is greater than 30 days.
+This section applies to MRS 3.2.0 or later.
+Description
MonitorServer checks whether its certificate file is about to expire every hour. This alarm is generated when the remaining validity period is at most 30 days. This alarm is automatically cleared when the remaining validity period is greater than 30 days.
Attribute
-
Alarm ID
+
- - Alarm ID
Alarm Severity
+- Alarm Severity
Auto Clear
+Auto Clear
24014
+@@ -24,30 +25,30 @@ - 24014
Major
+- Major
Yes
+Yes
Parameters
-
Name
+
- - Name
Meaning
+Meaning
Source
+- - Source
Specifies the cluster for which the alarm is generated.
+Specifies the cluster for which the alarm is generated.
ServiceName
+- - ServiceName
Specifies the service for which the alarm is generated.
+Specifies the service for which the alarm is generated.
RoleName
+- - RoleName
Specifies the role for which the alarm is generated.
+Specifies the role for which the alarm is generated.
HostName
+@@ -70,7 +71,7 @@ - HostName
Specifies the host for which the alarm is generated.
+Specifies the host for which the alarm is generated.
- Check whether this alarm is generated again during periodic system check.
- If yes, go to 10.
- If no, no further action is required.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select MonitorServer in the required cluster for Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select MonitorServer in the required cluster for Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-24015.html b/docs/mrs/umn/ALM-24015.html index 2217fa33..72a2064f 100644 --- a/docs/mrs/umn/ALM-24015.html +++ b/docs/mrs/umn/ALM-24015.html @@ -1,22 +1,23 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
ALM-24015 Flume MonitorServer Certificate File Has Expired
-Description
MonitorServer checks whether its certificate file in the system has expired every hour. This alarm is generated when the server certificate has expired. This alarm is automatically cleared when the MonitorServer certificate file becomes valid again.
+This section applies to MRS 3.2.0 or later.
+Description
MonitorServer checks whether its certificate file in the system has expired every hour. This alarm is generated when the server certificate has expired. This alarm is automatically cleared when the MonitorServer certificate file becomes valid again.
Attribute
-
Alarm ID
+
- - Alarm ID
Alarm Severity
+- Alarm Severity
Auto Clear
+Auto Clear
24015
+@@ -24,30 +25,30 @@ - 24015
Major
+- Major
Yes
+Yes
Parameters
-
Name
+
- - Name
Meaning
+Meaning
Source
+- - Source
Specifies the cluster for which the alarm is generated.
+Specifies the cluster for which the alarm is generated.
ServiceName
+- - ServiceName
Specifies the service for which the alarm is generated.
+Specifies the service for which the alarm is generated.
RoleName
+- - RoleName
Specifies the role for which the alarm is generated.
+Specifies the role for which the alarm is generated.
HostName
+@@ -71,7 +72,7 @@ - HostName
Specifies the host for which the alarm is generated.
+Specifies the host for which the alarm is generated.
- Check whether this alarm is generated again during periodic system check.
- If yes, go to 9.
- If no, no further action is required.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select MonitorServer in the required cluster for Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Select MonitorServer in the required cluster for Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-25000.html b/docs/mrs/umn/ALM-25000.html index de8d34dc..2b6bee65 100644 --- a/docs/mrs/umn/ALM-25000.html +++ b/docs/mrs/umn/ALM-25000.html @@ -70,7 +70,7 @@Alarm Clearing
This alarm is automatically cleared after the fault is rectified.
- Handle the alarm according to "ALM-12007 Process Fault".
- Check whether LdapServer Service Unavailable is cleared in the alarm list.
- If yes, no further action is required.
- If no, go to 10.
Collect fault information.
-+
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select LdapServer in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select LdapServer in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-25004.html b/docs/mrs/umn/ALM-25004.html index c41a8021..58a809d1 100644 --- a/docs/mrs/umn/ALM-25004.html +++ b/docs/mrs/umn/ALM-25004.html @@ -78,7 +78,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Check whether alarm Abnormal LdapServer Data Synchronization is cleared.
- If yes, no further action is required.
- If no, go to 15.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select LdapServer in the required cluster and OmsLdapServer from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select LdapServer in the required cluster and OmsLdapServer from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-25005.html b/docs/mrs/umn/ALM-25005.html index f5aa6eab..7f8ef6f4 100644 --- a/docs/mrs/umn/ALM-25005.html +++ b/docs/mrs/umn/ALM-25005.html @@ -60,7 +60,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Wait for 5 minutes and run the ps -ef | grep nscd command again as user root. Check whether the service exists.
Check whether the network is faulty, and whether the LDAP server can be accessed.
-
- Log in to the alarmed node as user root and run the ping command to check whether the network connectivity between this node and the LdapServer node. is normal.
+
- If yes, go to 6.
- If no, contact network administrators to troubleshoot the fault.
- Log in to the alarmed node as user root and run the ping command to check whether the network connectivity between this node and the LdapServer node is normal.
- If yes, go to 6.
- If no, contact network administrators to troubleshoot the fault.
Check whether the NameService is normal.
- Log in to the alarmed node as user root. Run the cat /etc/nsswitch.conf command to check whether the passwd, group, services, netgroup, and aliases of NameService are correctly configured.
The correct parameter configurations are as follows:
@@ -87,7 +87,7 @@ Feb 11 11:44:44 10-120-205-33 ntpq: nss_ldap: failed to bind to LDAP server ldap
- If yes, go to 15.
- If no, log in to other faulty nodes and repeat 12 to 14 to check whether the first LdapServer node in the URI before modifying /etc/ldap.conf is faulty. For example, check whether the service IP address is unreachable, the network delay is too long, or other abnormal software is deployed.
Collect the fault information.
-+
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select LdapClient for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select LdapClient for the target cluster.
- Click
- Contact O&M personnel and provide the collected logs.
diff --git a/docs/mrs/umn/ALM-25006.html b/docs/mrs/umn/ALM-25006.html index 87428b6a..c7b294d2 100644 --- a/docs/mrs/umn/ALM-25006.html +++ b/docs/mrs/umn/ALM-25006.html @@ -89,7 +89,7 @@ Feb 7 11:38:16 10-132-190-105 sssd[pam]: Starting upAlarm Clearing
This alarm is automatically cleared after the fault is rectified.
- If yes, go to 13.
- If no, log in to other faulty nodes and run 10 to 12. Collect logs and check whether the first ldapserver node in the ldap_uri before modifying /etc/sssd/sssd.conf is faulty. For example, check whether the service IP address is unreachable, the network latency is too long, or other abnormal software is deployed.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select LdapClient in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select LdapClient in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected fault logs.
diff --git a/docs/mrs/umn/ALM-25500.html b/docs/mrs/umn/ALM-25500.html index 64a5b31b..45b7a163 100644 --- a/docs/mrs/umn/ALM-25500.html +++ b/docs/mrs/umn/ALM-25500.html @@ -69,7 +69,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Handle the alarm according to "ALM-12004 OLdap Resource Abnormal".
- Check whether KrbServer Service Unavailable is cleared in the alarm list.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
-+
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select KrbServer in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select KrbServer in the required cluster from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-26051.html b/docs/mrs/umn/ALM-26051.html index 27d6e6c4..ee4a1fa7 100644 --- a/docs/mrs/umn/ALM-26051.html +++ b/docs/mrs/umn/ALM-26051.html @@ -77,7 +77,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
+
- On the FusionInsight Manager, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service drop-down list:
-
- KrbServer
KrbServer logs do not need to be downloaded in normal mode.
- ZooKeeper
- Storm
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-26052.html b/docs/mrs/umn/ALM-26052.html index f56acd41..a4e5e406 100644 --- a/docs/mrs/umn/ALM-26052.html +++ b/docs/mrs/umn/ALM-26052.html @@ -73,7 +73,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Storm and ZooKeeper in the required cluster from the Service drop-down list box.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Storm and ZooKeeper in the required cluster from the Service drop-down list box.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-26053.html b/docs/mrs/umn/ALM-26053.html index 2a412f34..7332526d 100644 --- a/docs/mrs/umn/ALM-26053.html +++ b/docs/mrs/umn/ALM-26053.html @@ -77,7 +77,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Storm and ZooKeeper in the required cluster from the Service drop-down list box.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select Storm and ZooKeeper in the required cluster from the Service drop-down list box.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-26054.html b/docs/mrs/umn/ALM-26054.html index 1507de36..64bf6aa2 100644 --- a/docs/mrs/umn/ALM-26054.html +++ b/docs/mrs/umn/ALM-26054.html @@ -73,7 +73,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following node in the required cluster from the Service drop-down list.
-
- NodeAgent
- Storm
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-27001.html b/docs/mrs/umn/ALM-27001.html index 7c94a655..7772db4e 100644 --- a/docs/mrs/umn/ALM-27001.html +++ b/docs/mrs/umn/ALM-27001.html @@ -90,7 +90,7 @@ NodeName ResName ResStatus ResHASAlarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Wait for about 2 minutes and check whether the alarm is cleared in the alarm list.
- If yes, no further action is required.
- If no, go to 19.
Collect fault information.
-+
- On FusionInsight Manager, choose O&M > Log > Download.
- Select DBService in the required cluster and NodeAgent from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On FusionInsight Manager, choose O&M > Log > Download.
- Select DBService in the required cluster and NodeAgent from the Service.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-27003.html b/docs/mrs/umn/ALM-27003.html index 31f4d05f..789db304 100644 --- a/docs/mrs/umn/ALM-27003.html +++ b/docs/mrs/umn/ALM-27003.html @@ -70,13 +70,13 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
Possible Causes
The link between the active and standby DBService nodes is abnormal.
Procedure
Check whether the network between the active DBService server and the standby DBService server is normal.
-
- In the alarm list on FusionInsight Manager, click
in the row where the alarm is located in the real-time alarm list and view the standby DBService server address.
- Log in to the active DBService server as user root.
- Run the ping standby DBService heartbeat IP address command to check whether the standby DBService server is reachable. +
- In the alarm list on FusionInsight Manager, click
in the row where the alarm is located in the real-time alarm list and view the standby DBService server address.
- Log in to the active DBService server as user root.
- Run the ping standby DBService heartbeat IP address command to check whether the standby DBService server is reachable.
- Contact the network administrator to check whether the network is faulty.
- Rectify the network fault and check whether the alarm is cleared from the alarm list.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- Select the following nodes in the required cluster from the Service:
-
- DBService
- Controller
- NodeAgent
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-27004.html b/docs/mrs/umn/ALM-27004.html index bb01548f..8fd07453 100644 --- a/docs/mrs/umn/ALM-27004.html +++ b/docs/mrs/umn/ALM-27004.html @@ -92,7 +92,7 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
- Expand the disk capacity.
- After the disk capacity is expanded, wait 2 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 16.
Collect fault information.
-+
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- In the Service area, select DBService of the target cluster and OS, OS Statistics, and OS Performance under OMS, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
- On the FusionInsight Manager portal, choose O&M > Log > Download.
- In the Service area, select DBService of the target cluster and OS, OS Statistics, and OS Performance under OMS, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact the O&M personnel and send the collected logs.
diff --git a/docs/mrs/umn/ALM-27005.html b/docs/mrs/umn/ALM-27005.html index ea963e28..b977b561 100644 --- a/docs/mrs/umn/ALM-27005.html +++ b/docs/mrs/umn/ALM-27005.html @@ -67,11 +67,11 @@Alarm Clearing
After the fault is rectified, the system automatically clears this alarm.
-Procedure
Checking whether too many data connections are used
- On FusionInsight Manager, click DBService in the service list on the left navigation pane. The DBService monitoring page is displayed.
- Observe the number of connections used by the database user, as shown in Figure 1. Based on the service scenario, reduce the number of database user connections. +
- On FusionInsight Manager, click DBService in the service list on the left navigation pane. The DBService monitoring page is displayed.
- Observe the number of connections used by the database user, as shown in Figure 1. Based on the service scenario, reduce the number of database user connections.
- Wait for 2 minutes and check whether the alarm is automatically cleared.
- If it is, no further action is required.
- If it is not, go to 4.
Checking whether the maximum number of database connections is properly configured
-
- Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > DBService > Configurations. On the displayed page, select the All Configurations tab, and increase the maximum number of database connections based on service requirements, as shown in Figure 2. Click Save. In the displayed Save configuration dialog box, click OK. +
- Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > DBService > Configurations. On the displayed page, select the All Configurations tab, and increase the maximum number of database connections based on service requirements, as shown in Figure 2. Click Save. In the displayed Save configuration dialog box, click OK.
- After the maximum number of database connections is changed, restart DBService (do not restart the upper-layer services).
Procedure: Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > DBService. On the displayed page, choose More > Restart Service. Enter the password of the current login user and click OK. Do not select Restart upper-layer services., click OK.
@@ -79,13 +79,13 @@
- After the service is restarted, wait for 2 minutes and check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 7.
- Log in to FusionInsight Manager and change the alarm threshold and alarm trigger count based on the actual database connection usage.
Choose O&M> Alarm > Thresholds > Name of the desired cluster > DBService > Database > Database Connections Usage (DBServer). In the Database Connections Usage (DBServer) area, click the pencil icon next to Trigger Count. In the displayed dialog box, change the trigger count, as shown in Figure 3.
