Description
The system checks the partition status every 60 seconds. This alarm is generated when the system detects that a partition to which service directories are mounted is lost (because the device is removed or goes offline, or the partition is deleted). The system checks the partition status periodically.
-This alarm must be manually cleared.
Attribute
Alarm ID @@ -17,7 +16,7 @@ | Major |
-No + |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
12033 |
-Minor + |
|
Yes |
diff --git a/docs/mrs/umn/ALM-12091.html b/docs/mrs/umn/ALM-12091.html
new file mode 100644
index 00000000..38437c04
--- /dev/null
+++ b/docs/mrs/umn/ALM-12091.html
@@ -0,0 +1,80 @@
+
+
+||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
12091 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
- The active/standby Manager switchover occurs.
- The disaster process restarts repeatedly, which may cause active/standby DR to be unavailable.
Possible Causes
The disaster process is abnormal.
+Handling Procedure
Check whether the disaster process is normal.
+- In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and click
to view the name of the host for which the alarm is generated. - Log in to the host for which the alarm is generated as user root.
- Run the su - omm command to switch to user omm.
- Run the sh ${BIGDATA_HOME}/om-server/OMS/workspace0/ha/module/hacom/script/status_ha.sh command to check whether the status of the disaster resources managed by the HA is normal. In the single-node system, the disaster resource is in the normal state. In the dual-node system, the disaster resource is in the normal state on the active node and in the stopped state on the standby node. +
- Run the vi ${BIGDATA_LOG_HOME}/disaster/disaster.log command to check whether the disaster resource log of HA contains the keyword ERROR. If yes, analyze the logs to locate the resource exception cause and fix the exception.
- Wait 5 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Disaster for the target cluster, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-12180 Suspended Disk I/O
-Description
- For HDDs, the alarm is triggered when any of the following conditions is met:
- The system collects data every 3 seconds, and detects that the svctm value exceeds 6s for 10 consecutive periods within 30 seconds.
- The system collects data every 3 seconds, and detects that the avgqu-sz value is greater than 0, the IOPS or bandwidth is 0, and the ioutil value is greater than 99% for 10 consecutive periods within 30 seconds.
Description
For MRS 3.3.0 and its later versions:
+- For HDDs, the alarm is triggered when any of the following conditions is met:
- By default, the system collects data every 3 seconds. The svctm latency reaches 6 seconds within 30 seconds in at least seven collection periods.
- By default, the system collects data every 3 seconds. The disk queue depth (avgqu-sz) is greater than 0, the IOPS or bandwidth is 0, and ioutil is greater than 99% in at least 10 collection periods within 30 seconds.
- By default, the system collects data every 3 seconds. At least 50% of detected svctm take no less than 1000 ms within 300 seconds.
- For SSDs, the alarm is triggered when any of the following conditions is met:
- By default, the system collects data every 3 seconds. The svctm latency reaches 3 seconds within 30 seconds in at least seven collection periods.
- By default, the system collects data every 3 seconds. The disk queue depth (avgqu-sz) is greater than 0, the IOPS or bandwidth is 0, and ioutil is greater than 99% in at least 10 collection periods within 30 seconds.
- By default, the system collects data every 3 seconds. At least 50% of detected svctm take no less than 500 ms within 300 seconds.
The collection period is 3 seconds, and the detection period is 30 or 300 seconds. This alarm is automatically cleared when neither of the preceding conditions is met for three consecutive detection periods (30 or 300 seconds).
+For versions earlier than MRS 3.3.0:
+- For HDDs, the alarm is triggered when any of the following conditions is met:
- The system collects data every 3 seconds, and detects that the svctm value exceeds 6s for 10 consecutive periods within 30 seconds.
- The system collects data every 3 seconds, and detects that the avgqu-sz value is greater than 0, the IOPS or bandwidth is 0, and the ioutil value is greater than 99% for 10 consecutive periods within 30 seconds.
- For SSDs, the alarm is triggered when any of the following conditions is met:
- The system collects data every 3 seconds, and detects that the svctm value exceeds 2s for 10 consecutive periods within 30 seconds.
- The system collects data every 3 seconds, and detects that the avgqu-sz value is greater than 0, the IOPS or bandwidth is 0, and the ioutil value is greater than 99% for 10 consecutive periods within 30 seconds.
This alarm is automatically cleared when the preceding conditions have not been met for 90s.
@@ -15,16 +21,17 @@- MRS 3.1.0:
Run the iostat -x -t command in the OS.
- Calculate svctm as follows in versions later than MRS 3.1.0:
svctm = (tot_ticks_new - tot_ticks_old)/(rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old)
-If rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old is 0, then svctm is 0.
-The parameters can be obtained as follows:
-The system runs the cat /proc/diskstats command every 3 seconds to collect data. For example:
-
-
In these two commands:
-In the data collected for the first time, the number in the fourth column is the rd_ios_old value, the number in the eighth column is the wr_ios_old value, and the number in the thirteenth column is the tot_ticks_old value.
-In the data collected for the second time, the number in the fourth column is the rd_ios_new value, the number in the eighth column is the wr_ios_new value, and the number in the thirteenth column is the tot_ticks_new value.
-In this case, the value of svctm is as follows:
-(19571460 - 19569526)/(1101553 + 28747977 - 1101553 - 28744856) = 0.6197
+- Versions earlier than MRS 3.3.0: If rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old = 0, then svctm = 0.
- MRS 3.3.0 and its later versions:
+When the detection period is 30 seconds, if rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old = 0, then svctm = 0.
+When the detection period is 300 seconds and rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old = 0, if tot_ticks_new - tot_ticks_old = 0, then svctm = 0; otherwise, the value of svctm is infinite.
The parameters can be obtained as follows:
+The system runs the cat /proc/diskstats command every 3 seconds to collect data. For example:
+
+In these two commands:
+In the data collected for the first time, the number in the fourth column is the rd_ios_old value, the number in the eighth column is the wr_ios_old value, and the number in the thirteenth column is the tot_ticks_old value.
+In the data collected for the second time, the number in the fourth column is the rd_ios_new value, the number in the eighth column is the wr_ios_new value, and the number in the thirteenth column is the tot_ticks_new value.
+In this case, the value of svctm is as follows:
+(19571460 - 19569526)/(1101553 + 28747977 - 1101553 - 28744856) = 0.6197
Attribute
diff --git a/docs/mrs/umn/ALM-12186.html b/docs/mrs/umn/ALM-12186.html new file mode 100644 index 00000000..a4b4a375 --- /dev/null +++ b/docs/mrs/umn/ALM-12186.html @@ -0,0 +1,83 @@ + + +ALM-12186 CGroup Task Usage Exceeds the Threshold
+Alarm Description
The system checks the CGroup task usage of user omm every 5 minutes. This alarm is generated when the CGroup task usage exceeds 90%. This alarm is cleared when the CGroup task usage is less than or equal to 90%.
+CGroup task usage = Number of used CGroup tasks/Maximum number of CGroup tasks
+You can run the systemctl status user-$(id -u).slice | grep limit | awk -F ' ' '{print $2}' command as user omm to obtain the number of used CGroup tasks of this user and run the echo $(systemctl status user-$(id -u).slice | grep limit | awk -F ' ' '{print $4}') | sed -e 's/)//g' command to obtain the maximum number of CGroup tasks allowed for this user.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
12186 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
- Failed to switch to user omm.
- Failed to create new omm processes.
- A faulty service or process cannot be restarted.
Possible Causes
The CGroup task usage exceeds 90%.
+Handling Procedure
Check the maximum number of threads that can be concurrently opened by user omm is properly set.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. On the page that is displayed, click
in the row containing the alarm, and view the name of the host for which the alarm is generated in Location. Click the host name to view its IP address. - Log in to the host for which the alarm is generated as user omm.
- Run the following command to obtain the maximum number of threads that can be concurrently opened by user omm and check whether this number is greater than or equal to 60000:
systemctl status user-$(id -u).slice | grep limit
+ + - Switch to user root and run the following command to change the value for user omm to 60000:
systemctl set-property user-2000.slice TasksMax=60000
+ - Change the value of UserTasksMax in the /etc/systemd/logind.conf file to 60000. (If the parameter is commented out, uncomment it.) Save the file, wait 5 minutes, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
+- On FusionInsight Manager of the cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select OmmServer and NodeAgent for the target cluster, and click OK.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-12187 Failed to Expand Disk Partition Capacity
+Alarm Description
The system checks the disk space every 60 seconds. When detecting that the disk space is expanded, the system expands disk partition. This alarm is generated when the disk partition fails to be expanded.
+This alarm is cleared when the system detects that the disk partition is successfully expanded.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
12187 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
MountDirectoryName + |
+Specifies the directory for which the alarm is generated. + |
+
Impact on the System
The expanded data disk space cannot be used to store data.
+Possible Causes
- The growpart scale-out tool is not installed.
- The system fails to execute the command for expanding disk partition.
Handling Procedure
Check whether growpart is installed.
+- Log in to FusionInsight Manager, click O&M, and choose Alarm > Alarms to view the alarm details. In the Location column, check the name of the host and mount directory for which the alarm is generated. Click the host name to view its IP address.
- Log in to the node for which the alarm is generated as user root.
- Run the following command to check whether growpart is installed:
which growpart
+If information similar to the following is displayed, the growpart tool is installed. Otherwise, contact O&M personnel to install the growpart tool.+[root@xxx ~]#which growpart +/usr/bin/growpart
+
- Wait for 5 minutes, then choose O&M, and choose Alarm > Alarms on FusionInsight Manager. Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Run the disk partition expansion command.
+- Run the following command to view the disk and partition information:
lsblk
+Search for the partition and the disk based on the mount directory name in the alarm location information, and check the disk and partition sizes.
+In the following example, the mount directory is /srv/BigData/data1, the used disk is /dev/vdb, and the disk partition is /dev/vdb1.
+
+
- Run the following command to expand the partition using growpart:
growpart Data disk Partition number
+Run the following command:
+growpart /dev/vdb 1
+If information similar to the following is displayed, the execution is successful. If the execution fails, contact O&M personnel.
+
+
- Run the following command to expand the file system size of the disk partition:
resize2fs Disk partition
+Run the following command:
+resize2fs /dev/vdb1
+If information similar to the following is displayed, the execution is successful:
+
+
- Wait for 5 minutes, click O&M, and choose Alarm > Alarms on FusionInsight Manager. Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, contact O&M personnel.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-12188 diskmgt Disk Monitoring Unavailable
+Alarm Description
NodeAgent checks the status of the diskmgt disk monitoring service every 5 minutes. This alarm is generated when diskmgt disk monitoring is unavailable.
+This alarm is cleared when the diskmgt disk monitoring service recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
12188 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
When diskmgt disk monitoring is unavailable, the read-only detection of the device partition file system, device partition loss detection, and disk partition scale-out detection cannot be performed.
+Possible Causes
- The diskmgt disk monitoring service does not exist.
- The diskmgt disk monitoring service is not started.
Handling Procedure
Check whether the diskmgt disk monitoring service exists.
+- Log in to FusionInsight Manager, click O&M, and choose Alarm > Alarms to view the alarm details. In the Location column, check the name of the host for which the alarm is generated. Click the host name to view its IP address.
- Log in to the node for which the alarm is generated as user root.
- Run the following command to check whether the core service file exists:
stat /usr/local/diskmgt/inner/diskmgtd
+If the file does not exist, contact O&M personnel.
+
Start the diskmgt disk monitoring service.
+- Run the following command to start the diskmgt disk monitoring service:
systemctl restart diskmgt
+ - Run the following command to check whether the diskmgt disk monitoring service is started:
systemctl status diskmgt
+- If information similar to the following is displayed, the service is started successfully. Go to 6.
+
- If no, contact O&M personnel.
- If information similar to the following is displayed, the service is started successfully. Go to 6.
- Wait for 5 minutes, click O&M, and choose Alarm > Alarms on FusionInsight Manager. Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, contact O&M personnel.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-14031 DataNode Process Is Abnormal
+Alarm Description
The DataNode process checks the process status every 20 seconds. This alarm is generated when the process status is abnormal and does not recover for a long time.
+This alarm is cleared when the process status recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
14031 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
If the process status is abnormal, the process cannot provide services properly. As a result, the entire service may become abnormal.
+Possible Causes
The host responds slowly to I/O (disk I/O and network I/O) requests and some processes are in the D state and Z state. The process may also be suspended and enter the T state.
+Handling Procedure
Check whether the process is in the D, Z, or T state.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. Wait for about 10 minutes and check whether the alarm is automatically cleared.
- If the alarm is not in the list, no further action is required.
- If the alarm is in the list, view the alarm details and record the IP address of the host where the alarm is generated. Run the command in 2.
- Log in to the host where the alarm is generated as the root user and run the su - omm command to switch to the omm user.
- Run the following command to check the process state:
ps ww -eo stat,cmd| grep -w org.apache.hadoop.hdfs.server.datanode.DataNode | grep -v grep | awk '{print$1}'
+ - Check whether the command output contains any abnormal state (D, Z, or T). +
- Switch to user root and run the reboot command to restart the host for which the alarm is generated. (Restarting a host is risky. Ensure that the service process is normal after the restart.)
- Wait 5 minutes and check whether the alarm is cleared.
- If the alarm is cleared, no further action is required.
- If the alarm fails to be cleared, go to 7.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-14032 JournalNode Process Is Abnormal
+Alarm Description
The JournalNode process checks the process status every 20 seconds. This alarm is generated when the process status is abnormal and does not recover for a long time.
+This alarm is cleared when the process status recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
14032 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
If the process status is abnormal, the process cannot provide services properly. As a result, the entire service may become abnormal.
+Possible Causes
The host responds slowly to I/O (disk I/O and network I/O) requests and some processes are in the D state and Z state. The process may also be suspended and enter the T state.
+Handling Procedure
Check whether the process is in the D, Z, or T state.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. Wait for about 10 minutes and check whether the alarm is automatically cleared.
- If the alarm is not in the list, no further action is required.
- If the alarm is in the list, view the alarm details and record the IP address of the host where the alarm is generated. Run the command in 2.
- Log in to the host where the alarm is generated as the root user and run the su - omm command to switch to the omm user.
- Run the following command to check the process state:
ps ww -eo stat,cmd| grep -w org.apache.hadoop.hdfs.qjournal.server.JournalNode | grep -v grep | awk '{print$1}'
+ - Check whether the command output contains any abnormal state (D, Z, or T). +
- Switch to user root and run the reboot command to restart the host for which the alarm is generated. (Restarting a host is risky. Ensure that the service process is normal after the restart.)
- Wait 5 minutes and check whether the alarm is cleared.
- If the alarm is cleared, no further action is required.
- If the alarm fails to be cleared, go to 7.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-14033 ZKFC Process Is Abnormal
+Alarm Description
The ZKFC process checks the process status every 20 seconds. This alarm is generated when the process status is abnormal and does not recover for a long time.
+This alarm is cleared when the process status recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
14033 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
If the process status is abnormal, the process cannot provide services properly. As a result, the entire service may become abnormal.
+Possible Causes
The host responds slowly to I/O (disk I/O and network I/O) requests and some processes are in the D state and Z state. The process may also be suspended and enter the T state.
+Handling Procedure
Check whether the process is in the D, Z, or T state.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. Wait for about 10 minutes and check whether the alarm is automatically cleared.
- If the alarm is not in the list, no further action is required.
- If the alarm is in the list, view the alarm details and record the IP address of the host where the alarm is generated. Run the command in 2.
- Log in to the host where the alarm is generated as the root user and run the su - omm command to switch to the omm user.
- Run the following command to check whether the process state is abnormal:
ps ww -eo stat,cmd| grep -w org.apache.hadoop.hdfs.tools.DFSZKFailoverController | grep -v grep | awk '{print$1}'
+ - Check whether the command output contains any abnormal state (D, Z, or T). +
- Switch to user root and run the reboot command to restart the host for which the alarm is generated. (Restarting a host is risky. Ensure that the service process is normal after the restart.)
- Wait 5 minutes and check whether the alarm is cleared.
- If the alarm is cleared, no further action is required.
- If the alarm fails to be cleared, go to 7.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-14034 Router Process Is Abnormal
+Alarm Description
The Router process checks the process status every 20 seconds. This alarm is generated when the process status is abnormal and does not recover for a long time.
+This alarm is cleared when the process status recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
14034 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
If the process status is abnormal, the process cannot provide services properly. As a result, the entire service may become abnormal.
+Possible Causes
The host responds slowly to I/O (disk I/O and network I/O) requests and some processes are in the D state and Z state. The process may also be suspended and enter the T state.
+Handling Procedure
Check whether the process is in the D, Z, or T state.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. Wait for about 10 minutes and check whether the alarm is automatically cleared.
- If the alarm is not in the list, no further action is required.
- If the alarm is in the list, view the alarm details and record the IP address of the host where the alarm is generated. Run the command in 2.
- Log in to the host where the alarm is generated as the root user and run the su - omm command to switch to the omm user.
- Run the following command to check whether the process state is abnormal:
ps ww -eo stat,cmd| grep -w org.apache.hadoop.hdfs.server.federation.router.DFSRouter | grep -v grep | awk '{print$1}'
+ - Check whether the command output contains any abnormal state (D, Z, or T). +
- Switch to user root and run the reboot command to restart the host for which the alarm is generated. (Restarting a host is risky. Ensure that the service process is normal after the restart.)
- Wait 5 minutes and check whether the alarm is cleared.
- If the alarm is cleared, no further action is required.
- If the alarm fails to be cleared, go to 7.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-14035 HttpFS Process Is Abnormal
+Alarm Description
The HttpFS process checks the process status every 20 seconds. This alarm is generated when the process status is abnormal and does not recover for a long time.
+This alarm is cleared when the process status recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
14035 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
If the process status is abnormal, the process cannot provide services properly. As a result, the entire service may become abnormal.
+Possible Causes
The host responds slowly to I/O (disk I/O and network I/O) requests and some processes are in the D state and Z state. The process may also be suspended and enter the T state.
+Handling Procedure
Check whether the process is in the D, Z, or T state.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. Wait for about 10 minutes and check whether the alarm is automatically cleared.
- If the alarm is not in the list, no further action is required.
- If the alarm is in the list, view the alarm details and record the IP address of the host where the alarm is generated. Run the command in 2.
- Log in to the host where the alarm is generated as the root user and run the su - omm command to switch to the omm user.
- Run the following command to check whether the process state is abnormal:
ps ww -eo stat,cmd| grep -w org.apache.hadoop.fs.http.server.HttpFSServerWebServer | grep -v grep | awk '{print$1}'
+ - Check whether the command output contains any abnormal state (D, Z, or T). +
- Switch to user root and run the reboot command to restart the host for which the alarm is generated. (Restarting a host is risky. Ensure that the service process is normal after the restart.)
- Wait 5 minutes and check whether the alarm is cleared.
- If the alarm is cleared, no further action is required.
- If the alarm fails to be cleared, go to 7.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-19022 HBase Hotspot Detection Is Unavailable
+Alarm Description
When the MetricController instance is installed for HBase, the alarm module checks the health status of the active HBase MetricController instance every 120 seconds. This alarm is generated when the active HBase MetricController instance does not exist or is unavailable and the hotspot detection function is unavailable.
+This alarm is cleared when the active HBase MetricController instance recovers.
+
This alarm applies only to MRS 3.3.0 or later.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
19022 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
The HBase hotspot detection function is unavailable.
+Possible Causes
- The ZooKeeper service is abnormal.
- The HBase service is abnormal.
- In the current HBase service, the MetricController instance on the same node as the active HMaster instance is not started.
- The network is abnormal.
Handling Procedure
Check the ZooKeeper service status.
+- In the service list on FusionInsight Manager, check whether Running Status of ZooKeeper is Normal. +
- In the alarm list, check whether ALM-13000 ZooKeeper Service Unavailable exists. +
- Rectify the fault by performing the operations provided for ALM-13000 ZooKeeper Service Unavailable.
- Wait for several minutes and check whether the alarm HBase Hotspot Detection Is Unavailable is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check the HBase service status.
+- In the service list on FusionInsight Manager, check whether Running Status of HBase is Normal. +
- In the alarm list, check whether the alarm ALM-19000 HBase Service Unavailable exists. +
- Rectify the fault by following the steps provided for ALM-19000 HBase Service Unavailable.
- Wait for several minutes and check whether the alarm HBase Hotspot Detection Is Unavailable is cleared.
- If yes, no further action is required.
- If no, go to 9.
Check whether the MetricController instance deployed on the same node as the active HMaster instance is started.
+- On FusionInsight Manager, choose Cluster > Service > HBase, and click Instances to check whether the MetricController(Active) instance exists. +
- Select the MetricController instance whose management IP address is the same as that of the active HMaster instance, and click Start Instance.
- After the MetricController instance is restarted, check whether the alarm HBase Hotspot Detection Is Unavailable is cleared.
- If yes, no further action is required.
- If no, go to 12.
Check the network connectivity between the started MetricController instances and the active HMaster node.
+- Log in to the node where the active HMaser instance is deployed and run ping IP address of the node where the standby MetricController instance is deployed to check whether the network connection between the started MetricController instances and the host where the active HMaster instance is deployed is normal. +
- Contact the network administrator to restore the network.
- After the network recovers, check whether the alarm HBase Hotspot Detection Is Unavailable is cleared.
- If yes, no further action is required.
- If no, go to 15.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- In the Host area, select the host where the HMaster instance is deployed.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-19023 Region Traffic Restriction for HBase
+Alarm Description
When the MetricController instance is installed for the HBase service, self-healing from hotspotting is automatically enabled. The alarm module checks whether there are regions whose request traffic is restricted due to hotspot issues in HBase every 120 seconds. This alarm is generated when the region where hotspot traffic is restricted is detected in HBase.
+This alarm is cleared when the region is no longer a hotspot.
+This alarm applies only to MRS 3.3.0 or later.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
19023 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
If the traffic of a hotspot region is restricted, the number of handlers for processing the requests in the region is limited. As a result, services requesting the region may slow down or retry upon failure.
+Possible Causes
Too many requests are directed to a single region when the HBase service is accessed.
+Handling Procedure
Check whether there are too many requests in a single region of HBase.
+- Log in to FusionInsight Manager, and Choose O&M > Alarm > Alarms.
- In Additional Information of Region Traffic Restriction for HBase, view the reported table name and region information.
- On FusionInsight Manager, choose Cluster > Service > HBase and click the hyperlink on the right of HMaster web UI.
- Click Table Details and adjust service configurations in the region where the table in 2 is deployed.
- Wait a moment and then check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- In the Host area, select the host where the HMaster instance is deployed.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm will be automatically cleared.
+Related Information
None.
+ALM-19024 RPC Requests P99 Latency on RegionServer Exceeds the Threshold
+Alarm Description
The system checks P99 latency for RPC requests on each RegionServer instance of the HBase service every 30 seconds. This alarm is generated when P99 latency for RPC requests on a RegionServer exceeds the threshold for 10 consecutive times.
+This alarm is cleared when P99 latency for RPC requests on a RegionServer instance is less than or equal to the threshold.
+This alarm applies only to MRS 3.3.0 or later.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
19024 + |
+
|
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
If RPC requests P99 latency exceeds the threshold, the RegionServer cannot deliver normal service performance externally. If RPC requests P99 latency on most RegionServers in the cluster exceeds the threshold, HBase may fail to provide services for external systems.
+Possible Causes
- RegionServer GC duration is too long.
- The HDFS RPC response is too slow.
- RegionServer request concurrency is too high.
Handling Procedure
- Log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Alarm > Alarms. On the page that is displayed, locate the row containing the alarm whose Alarm ID is 19024, and view the service instance and host name in Location.
Check the GC duration of RegionServer.
+- In the alarm list on FusionInsight Manager, check whether the "HBase GC Duration Exceeds the Threshold" alarm is generated for the service instance in 1. +
- Rectify the fault by following the handling procedure of "ALM-19007 HBase GC Duration Exceeds the Threshold".
- Wait several minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check HDFS RPC response time.
+- In the alarm list on FusionInsight Manager, check whether alarm "Average NameNode RPC Processing Time Exceeds the Threshold" is generated for the HDFS service on which the HBase service depends. +
- Rectify the fault by following the handling procedure of "ALM-14021 Average NameNode RPC Processing Time Exceeds the Threshold".
- Wait several minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Check the number of concurrent processes on a RegionServer.
+- In the alarm list on FusionInsight Manager, check whether the "Handler Usage of RegionServer Exceeds the Threshold" alarm is generated for the service instance in 1. +
- Rectify the fault by following the handling procedure of "ALM-19021 Handler Usage of RegionServer Exceeds the Threshold".
- Wait several minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-19025 Damaged StoreFile in HBase
+Alarm Description
The system checks the hdfs://hacluster/hbase/autocorrupt and hdfs://hacluster/hbase/MasterData/autocorrupt directories on HDFS of each HBase service every 120 seconds. This alarm is generated when there are files in the directories.
+This alarm is cleared when the hdfs://hacluster/hbase/autocorrupt and hdfs://hacluster/hbase/MasterData/autocorrupt directories do not exist or are empty.
+This alarm applies only to MRS 3.3.0 or later.
+ hdfs://hacluster indicates the name of the file system used by HBase, and /hbase indicates the root directory of HBase in the file system. You can log in to FusionInsight Manager, choose Cluster > Services > HBase and click Configuration. Search for fs.defaultFS and hbase.data.rootdir.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
19025 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
There are damaged StoreFile files in HBase, which may cause data loss.
+Possible Causes
The StoreFile files are damaged.
+Handling Procedure
- Log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Alarm > Alarms. On the page that is displayed, locate the row containing the alarm whose Alarm ID is 19025, and view the service in Location.
- Log in to the node where the HDFS and HBase clients are installed as the client installation user and run the following commands:
cd Client installation directory
+source bigdata_env
+kinit Component service user (If Kerberos authentication is disabled for the cluster (the cluster is in normal mode), skip this step.)
+ - Check the damaged StoreFile file.
- Run the following command to check whether the /hbase/autocorrupt directory of HDFS is empty. If it is not, go to 4.
hdfs dfs -ls -R hdfs://hacluster/hbase/autocorrupt
+ - Run the following command to check whether the /hbase/MasterData/autocorrupt directory of HDFS is empty. If it is not, go to 9.
hdfs dfs -ls -R hdfs://hacluster/hbase/MasterData/autocorrupt
+
- Run the following command to check whether the /hbase/autocorrupt directory of HDFS is empty. If it is not, go to 4.
- Run the following command to restore the StoreFile files in the hdfs://hacluster/hbase/autocorrupt directory:
hdfs debug recoverLease -path hdfs://hacluster/hbase/autocorrupt/Name space/Table/Region/Column family/StoreFile files
+ - Check whether the damaged StoreFile files are restored. If the following information is displayed, the restoration is successful:
recoverLease SUCCEEDED on hdfs://hacluster/hbase/autocorrupt/default/h1/865665fe32db62dadada68b644359809/cf1/95f210f931ad44c99e4028470be7d292
+If yes, go to 6.
+If no, go to 9.
+ - Run the following command to move the files back to the hdfs://hacluster/hbase/data directory:
hdfs dfs -mv hdfs://hacluster/hbase/autocorrupt/Name space/Table/Region/Column family/StoreFile fileshdfs://hacluster/hbase/data/Name space/Table/Region/Column family/StoreFile files
+ - Run the following command on HBase Shell to bring the region online again:
hbase shell
+unassign'Region'
+assign'Region'
+ - Wait several minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-19026 Damaged WAL Files in HBase
+Alarm Description
The system checks the hdfs://hacluster/hbase/corrupt directory on the HDFS of each HBase service every 120 seconds. This alarm is generated when there are WAL files in the /hbase/corrupt directory.
+This alarm is cleared when the /hbase/corrupt directory does not exist or does not contain WAL files.
+This alarm applies only to MRS 3.3.0 or later.
+ hdfs://hacluster indicates the name of the file system used by HBase, and /hbase indicates the root directory of HBase in the file system. You can log in to FusionInsight Manager, choose Cluster > Services > HBase and click Configuration. Search for fs.defaultFS and hbase.data.rootdir.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
19026 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
There are damaged WAL files in HBase, which may cause data loss.
+Possible Causes
The WAL files are damaged.
+Handling Procedure
- Log in to FusionInsight Manager and choose O&M. In the navigation pane on the left, choose Alarm > Alarms. On the page that is displayed, locate the row containing the alarm whose Alarm ID is 19026, and view the service in Location.
- Log in to the node where the HDFS clients are installed as the client installation user and run the following commands:
cd Client installation directory
+source bigdata_env
+kinit Component service user (If Kerberos authentication is disabled for the cluster (the cluster is in normal mode), skip this step.)
+ - Run the following command to check the damaged WAL files and go to 4:
hdfs dfs -ls hdfs://hacluster/hbase/corrupt/*%2C*
+
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select HBase for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-25007 Number of SlapdServer Connections Exceeds the Threshold
+Alarm Description
The system checks the number of process connections on the SlapdServer node every 30 seconds and compares the actual number with the threshold. This alarm is generated when the number of process connections exceeds the threshold (1000 by default) for multiple times (5 by default).
+Its Trigger Count is configurable. If Trigger Count is set to 1, this alarm is cleared when the number of process connections is less than or equal to the threshold. If Trigger Count is greater than 1, this alarm is cleared when the number of process connections is less than or equal to 90% of the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
25007 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Processes respond slowly or do not work.
+Possible Causes
- There are too many SlapdServer connections.
- The alarm threshold or alarm trigger count is improperly configured.
Handling Procedure
Check whether there are too many SlapdServer process connections.
+- Log in to FusionInsight Manager and choose Cluster > Services > LdapServer.
- On the LdapServer dashboard page, observe the SlapdServer process connections and decrease the connections based on service requirements.Figure 1 SlapdServer process connections+
- Wait about 2 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
Check whether the alarm threshold or alarm trigger count is properly configured.
+- On FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, choose LdapServer > Other > SlapdServer Service Connections, and check whether the alarm trigger count and alarm threshold are set properly. +
- Change the trigger count and alarm threshold based on the actual number of process connections, and apply the changes.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select LdapServer for the target cluster.
- Specify Hosts for collecting logs, which is optional. By default, all hosts are selected.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-25008 SlapdServer CPU Usage Exceeds the Threshold
+Alarm Description
The system checks the CPU usage of the SlapdServer node every 30 seconds and compares the actual usage with the threshold. This alarm is generated when the SlapdServer CPU usage exceeds the threshold for multiple times (5 by default).
+Its Trigger Count is configurable. If Trigger Count is set to 1, this alarm is cleared when the SlapdServer CPU usage is less than or equal to the threshold. If Trigger Count is greater than 1, this alarm is cleared when the SlapdServer CPU usage is less than or equal to 90% of the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
25008 + |
+Critical (default threshold: 85%) +Major (default threshold: 75%) + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Processes respond slowly or do not work.
+Possible Causes
- The alarm threshold or alarm trigger count is improperly configured.
- The CPU configuration cannot meet service requirements, and the CPU usage reaches the upper limit.
Handling Procedure
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, choose LdapServer > Other > SlapdServer Service Total CPU Percentage, and check whether the alarm trigger count and alarm threshold are set properly. +
- Change the trigger count and alarm threshold based on the actual CPU usage, and apply the changes.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 4.
Check whether the CPU usage reaches the upper limit.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms. In the right pane, click this alarm and obtain the host name in Location.
- Choose Cluster > Services > LdapServer, click the Instance tab, and click the SlapdServer instance corresponding to the host name in 4.
- On the dashboard of the instance, observe the real-time data of the CPU Usage of a Single SlapdServer Instance chart for about 5 minutes and check whether the CPU usage exceeds the threshold (75% by default) for multiple times. +
- Check whether the status of other SlapdServer instances is normal. For details, see 5 to 6. +
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select LdapServer for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-29007 Impalad Process Memory Usage Exceeds the Threshold
+Alarm Description
The system checks the memory usage of the Impalad process every 30 seconds. This alarm is generated when the system detects that the memory usage exceeds the default threshold (80%).
+This alarm is automatically cleared when the system detects that the memory usage of the process falls below the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
29007 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Type + |
+Parameter + |
+Description + |
+
|---|---|---|
Location Information + |
+Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+|
RoleName + |
+Specifies the role for which the alarm was generated. + |
+|
HostName + |
+Specifies the host for which the alarm was generated. + |
+|
Additional Information + |
+Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The memory usage is too high. Some query tasks may fail due to insufficient memory.
+Possible Causes
The Impalad process is executing a large number of query tasks.
+Handling Procedure
- On FusionInsight Manager, choose O&M > Alarm > Thresholds > Impala > CPU and Memory > Impalad Process Memory Usage (Impalad) and check the threshold.
- If the alarm threshold is smaller than 80%, increase the alarm threshold as required and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 3.
- If the threshold is greater than 80%, check whether a large number of concurrent query tasks exist when the alarm is generated. A large number of concurrent query tasks will cause the memory usage to increase sharply. After the tasks are complete, check whether the alarm is automatically cleared. During this period, some tasks may fail to be executed or may be canceled due to insufficient memory. In this case, try again.+
If the memory usage always exceeds the threshold, the cluster capacity needs to be expanded.
+- If yes, no further action is required.
- If no, go to 4.
Collect fault information.
+- On FusionInsight Manager of the active or standby cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Impala for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
The alarm is automatically cleared after the burst concurrent tasks are complete.
+Related Information
None
+ALM-29008 Number of ODBC Connections to Impalad Exceeds the Threshold
+Alarm Description
The system checks the number of client connections to the Impalad node every 30 seconds. This alarm is generated when the number of client connections exceeds the customized threshold (60 by default).
+This alarm is automatically cleared when the number of client connections is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
29008 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Type + |
+Parameter + |
+Description + |
+
|---|---|---|
Location Information + |
+Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+|
RoleName + |
+Specifies the role for which the alarm was generated. + |
+|
HostName + |
+Specifies the host for which the alarm was generated. + |
+|
Additional Information + |
+Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
New client connections may be blocked or even fail.
+Possible Causes
The number of client connections maintained by the Impalad service is too large or the threshold is too small.
+Handling Procedure
- On FusionInsight Manager, choose O&M > Alarm > Thresholds > Impala > Connections > Number of ODBC Connections to Impalad Process (Impalad) to check the threshold.
- Check the number of ODBC applications connected to Impalad and stop idle applications. Check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 3 to change the number of concurrent connections supported by Impalad.
- On FusionInsight Manager, choose Cluster > Impala > Configurations > All Configurations > Impalad > Customization. Add the custom parameter --fe_service_threads. The default value of this parameter is 64. Change the value as required and click Save.
- After the query tasks on all clients are complete, click the Instances tab. Select all Impalad instances, and restart them.+
The service will become unavailable when all instances are restarted. If a single instance is restarted, the tasks that are being executed on that instance will fail and the service will become available.
+ - After the restart is complete, check whether the alarm is cleared.
- If yes, no further action is required.
- If yes, go to 6.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Impala for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-29010 Number of Queries Being Submitted by Impalad Exceeds the Threshold
+Alarm Description
The system checks the total number of queries being submitted by the Impalad node every 60 seconds. This alarm is generated when the number of queries exceeds the customized threshold (150 by default).
+This alarm is automatically cleared when the number of queries is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
29010 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Type + |
+Parameter + |
+Description + |
+
|---|---|---|
Location Information + |
+Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+|
RoleName + |
+Specifies the role for which the alarm was generated. + |
+|
HostName + |
+Specifies the host for which the alarm was generated. + |
+|
Additional Information + |
+Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The queries may be blocked or even fail.
+Possible Causes
The Impalad service has maintained a large number of queries, or the threshold is too small.
+Handling Procedure
- On FusionInsight Manager, choose O&M > Alarm > Thresholds > Impala > Query Task Sum Statistics > Total number of Queries Being Submitted (Impalad) and check the threshold.
+
- Change the threshold.
+
- Click the Instances tab, select all Impalad instances, and restart them.+
The service will become unavailable when all instances are restarted. If a single instance is restarted, the tasks that are being executed on that instance will fail and the service will become available.
+
+
- After the restart is complete, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Impala for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-29011 Number of Queries Being Executed by Impalad Exceeds the Threshold
+Alarm Description
The system checks the total number of queries being executed by the Impalad node every 60 seconds. This alarm is generated when the number of queries exceeds the customized threshold (150 by default).
+This alarm is automatically cleared when the number of queries is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
29011 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Type + |
+Parameter + |
+Description + |
+
|---|---|---|
Location Information + |
+Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+|
RoleName + |
+Specifies the role for which the alarm was generated. + |
+|
HostName + |
+Specifies the host for which the alarm was generated. + |
+|
Additional Information + |
+Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The queries may be blocked or even fail.
+Possible Causes
The Impalad service has maintained a large number of queries, or the threshold is too small.
+Handling Procedure
- On FusionInsight Manager, choose O&M > Alarm > Thresholds > Impala > Query Task Sum Statistics > Total number of Queries Being Executed (Impalad) and check the threshold.
+
- Change the threshold.
+
- Click the Instances tab, select all Impalad instances, and restart them.+
The service will become unavailable when all instances are restarted. If a single instance is restarted, the tasks that are being executed on that instance will fail and the service will become available.
+
+
- After the restart is complete, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Impala for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-29012 Number of Queries Being Waited by Impalad Exceeds the Threshold
+Alarm Description
The system checks the total number of queries being waited by the Impalad node every 60 seconds. This alarm is generated when the number of queries exceeds the customized threshold (150 by default).
+This alarm is automatically cleared when the number of queries is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
29012 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Type + |
+Parameter + |
+Description + |
+
|---|---|---|
Location Information + |
+Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+|
RoleName + |
+Specifies the role for which the alarm was generated. + |
+|
HostName + |
+Specifies the host for which the alarm was generated. + |
+|
Additional Information + |
+Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The queries may be blocked or even fail.
+Possible Causes
The Impalad service has maintained a large number of queries, or the threshold is too small.
+Handling Procedure
- On FusionInsight Manager, choose O&M > Alarm > Thresholds > Impala > Query Task Sum Statistics > Total number of Waiting Queries (Impalad) and check the threshold.
+
- Change the threshold.
+
- Click the Instances tab, select all Impalad instances, and restart them.+
The service will become unavailable when all instances are restarted. If a single instance is restarted, the tasks that are being executed on that instance will fail and the service will become available.
+
+
- After the restart is complete, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Impala for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-29013 Impalad FGC Time Exceeds the Threshold
+Alarm Description
The system checks the FGC time of the Impalad service every 60 seconds. This alarm is generated when the FGC time exceeds the threshold (12 seconds) for five consecutive times. This alarm is cleared when the FGC time is less than or equal to the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
29013 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Type + |
+Parameter + |
+Description + |
+
|---|---|---|
Location Information + |
+Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+|
RoleName + |
+Specifies the role for which the alarm was generated. + |
+|
HostName + |
+Specifies the host for which the alarm was generated. + |
+|
Additional Information + |
+Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Data read and write are affected.
+Possible Causes
The memory of the node instance is overused or the heap memory is inappropriately allocated, causing frequent occurrence of the GC process.
+Handling Procedure
Check the GC time.
+- Choose O&M > Alarm > Thresholds > Impala > Process FGCT > Process FGCT of Impalad (Impalad), and check the threshold (12s by default).
+
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and check whether the alarm whose Alarm ID is 29013 exists in the alarm list.
- If yes, go to 3.
- If no, no further action is required.
- On FusionInsight Manager, choose Cluster > Impala, click the Instances tab, select the Impalad instance for which the alarm is generated, then click the Chart tab, locate the Process FGCT chart, and check whether the FGC time is greater than the threshold in 1. +
- Choose O&M > Alarm > Thresholds > Impala > Process FGCT > Process FGCT of Impalad (Impalad), and change the threshold to a value less than the time obtained in 3. Then, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager of the active or standby cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Impala for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-29014 Catalog FGC Time Exceeds the Threshold
+Alarm Description
The system checks the FGC time of the Catalog service every 60 seconds. This alarm is generated when the FGC time exceeds the threshold (12 seconds) for five consecutive times. This alarm is cleared when the FGC time is less than or equal to the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
29014 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Type + |
+Parameter + |
+Description + |
+
|---|---|---|
Location Information + |
+Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+|
RoleName + |
+Specifies the role for which the alarm was generated. + |
+|
HostName + |
+Specifies the host for which the alarm was generated. + |
+|
Additional Information + |
+Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Data read and write are affected.
+Possible Causes
The memory of the node instance is overused or the heap memory is inappropriately allocated, causing frequent occurrence of the GC process.
+Handling Procedure
Check the GC time.
+- Choose O&M > Alarm > Thresholds, click the name of the desired cluster, choose Impala > Process FGCT > Process FGCT of Catalog (Catalog), and check the threshold (12s by default).
+
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and check whether the alarm whose Alarm ID is 29014 exists in the alarm list.
- If yes, go to 3.
- If no, no further action is required.
- On FusionInsight Manager, choose Cluster > Impala, click the Instance tab, select the Catalog instance for which the alarm is generated, then click the Chart tab, locate the Process FGCT chart, and check whether the FGC time is greater than the threshold in 1. +
- Choose O&M > Alarm > Thresholds, click the name of the desired cluster, choose Impala > Process FGCT > Process FGCT of Catalog (Catalog), and change the threshold to a value less than the time obtained in 3. Then, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager of the active or standby cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Impala for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-29015 Catalog Process Memory Usage Exceeds the Threshold
+Alarm Description
The system checks the memory usage of the Catalog process every 30 seconds. This alarm is generated when the system detects that the memory usage exceeds the default threshold (80%).
+This alarm is automatically cleared when the system detects that the memory usage of the process falls below the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
29015 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Type + |
+Parameter + |
+Description + |
+
|---|---|---|
Location Information + |
+Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+|
RoleName + |
+Specifies the role for which the alarm was generated. + |
+|
HostName + |
+Specifies the host for which the alarm was generated. + |
+|
Additional Information + |
+Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The memory usage is too high. Some query tasks may fail due to insufficient memory.
+Possible Causes
The memory of the node instance is overused or the memory is inappropriately configured.
+Handling Procedure
- On FusionInsight Manager, choose O&M > Alarm > Thresholds > Impala > CPU and Memory > Catalog Process Memory Usage (Impalad) and check the threshold.
- If the alarm threshold is smaller than 80%, increase the alarm threshold as required and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 3.
- If the threshold is greater than 80%, check whether a large number of concurrent query tasks exist when the alarm is generated. A large number of concurrent query tasks will cause the memory usage to increase sharply. After the tasks are complete, check whether the alarm is automatically cleared. During this period, some tasks may fail to be executed or may be canceled due to insufficient memory. In this case, try again.+
If the memory usage always exceeds the threshold, the cluster capacity needs to be expanded.
+- If yes, no further action is required.
- If no, go to 4.
Collect fault information.
+- On FusionInsight Manager of the active or standby cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Impala for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-29016 Impalad Instance in the Sub-healthy State
+Alarm Description
In MRS 3.1.5, the system checks every 60 seconds whether the Hive Server2 HTTP port (28000) of Impalad responds to cURL requests. This alarm is generated when the returned result has been incorrect for 20 seconds in two consecutive times. This alarm is cleared when the system correctly responds within 20 seconds.
+In other MRS versions, the system checks every 60 seconds whether Impalad can execute select 1. This alarm is generated when the returned result has been incorrect for 20 seconds in two consecutive times. This alarm is cleared when the SQL statement is correctly executed within 20 seconds.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
29016 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Type + |
+Parameter + |
+Description + |
+
|---|---|---|
Location Information + |
+Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+|
RoleName + |
+Specifies the role for which the alarm was generated. + |
+|
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Impact on the System
Impalad cannot execute SQL statements or SQL statement execution times out, which affects data read and write.
+Possible Causes
The Impalad service maintains too many queries.
+Handling Procedure
- Log in to FusionInsight Manager and choose Cluster > Services > Impala > Impalad Web UI. On the displayed page, click any node to go to the web UI.
- On the web UI, click /backends to view the Impala instance list. Locate the instance for which the alarm is generated and click Web UI. After the web UI of the subhealthy node is displayed, click /queries to check the task execution status and check whether any task is executed slowly. +
- After the task is complete, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
- On FusionInsight Manager, choose Cluster > Services > Impala > Instances, select the Impala instance for which the alarm is generated, click More, and select Restart Instance. Then, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.+
The service will become unavailable when all instances are restarted. If a single instance is restarted, the tasks that are being executed on that instance will fail and the service will become available.
+
Collect fault information.
+- On FusionInsight Manager of the active or standby cluster, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Impala for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-45003 HetuEngine QAS Disk Capacity Is Insufficient
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the HetuEngine QAS disk usage every 60 seconds and compares the actual disk usage with the threshold. The disk usage has a default threshold. This alarm is generated if the disk usage exceeds the threshold.
+To change the threshold, choose O&M > Alarm > Thresholds. In the service list, choose HetuEngine > Disk > QAS Disk Usage (QAS).
+If the Trigger Count is 1, this alarm is cleared when the usage of the HetuEngine QAS disk is less than or equal to the threshold. If the Trigger Count is greater than 1, this alarm is cleared when the disk usage is less than or equal to 80% of the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45003 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
PartitionName + |
+Specifies the disk partition for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
If the disk capacity is insufficient, QAS fails to write data, affecting SQL diagnosis and automatic recommendation of materialized views.
+Possible Causes
- The alarm threshold is improperly configured.
- The configuration of the HetuEngine QAS disk cannot meet service requirements. The disk usage reaches the upper limit.
Handling Procedure
Check whether the threshold is set properly.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Thresholds. In the service list, choose HetuEngine > Disk > QAS Disk Usage (QAS). Check whether the alarm threshold is set properly. The default threshold is 80% of the disk capacity. You can change the threshold as required. +
- Click Modify in the Operation column to modify and save the alarm threshold as required.
- Wait 2 minutes and check whether the alarm is cleared.
- If the alarm is cleared, no further action is required.
- If the alarm is not cleared, go to 4.
Check whether the disk usage reaches the upper limit.
+- Expand the alarm information, view the information in the Location area, and check the role name and host name of the QAS disk where the alarm is generated.
- Choose Cluster > Services > HetuEngine and click Instance. On the displayed page, click the QAS role name in the alarm information. On the instance page that is displayed, click Chart and check whether the QAS disk usage in the QAS Disk Usage chart exceeds the threshold (80% of the disk capacity by default). +
- Log in to the host of the node where the QAS instance reporting the alarm is located as the root user.
- Run the following command to go to the QAS data directory and delete temporary files as required:
cd ${BIGDATA_DATA_HOME}/hetuengine/qas
++
Deleting temporary files affects the latest QAS execution result but does not affect subsequent results.
+ - Wait 2 minutes and check whether the alarm is cleared.
- If the alarm is cleared, no further action is required.
- If the alarm fails to be cleared, go to 9.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select HetuEngine for the target cluster, and click OK.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the hosts to which the role belongs, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45289 PolicySync Heap Memory Usage Exceeds the Threshold
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks the heap memory usage of the PolicySync service every 60 seconds. This alarm is generated when the heap memory usage of the PolicySync instance exceeds the threshold (95% of the maximum memory) for 10 consecutive times. This alarm is cleared when the heap memory usage is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45289 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Heap memory overflow may cause service breakdown.
+Possible Causes
The heap memory of the PolicySync instance is overused or the heap memory is inappropriately allocated.
+Handling Procedure
- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms > ALM-45289 PolicySync Heap Memory Usage Exceeds the Threshold. Check the location information of the alarm and view the host name of the instance for which the alarm is generated.
- On FusionInsight Manager, choose Cluster > Services > Ranger > Instance. Select the role corresponding to the host name of the instance for which the alarm is generated. Click the drop-down list in the upper right corner of the chart area and choose Customize > CPU and Memory > PolicySync Heap Memory Usage. Click OK.
- Check whether the heap memory used by PolicySync reaches the threshold (95% of the maximum heap memory by default). +
- On FusionInsight Manager, choose Cluster > Services > Ranger > Instance > PolicySync. Click Instance Configuration and then All Configurations, and choose PolicySync > System. Set -Xmx in the GC_OPTS parameter to a larger value based on site requirements and save the configuration.+
If this alarm is generated, the heap memory configured for PolicySync cannot meet the heap memory required by the PolicySync process. You are advised to change the value of -Xmx in GC_OPTS to twice that of the heap memory used by PolicySync. You can change the value based on the actual service scenario. Refer to 2 to view the PolicySync heap memory usage.
+ - Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.+
When the service is rebooted, it becomes unavailable and can disrupt business operations. When the instance is rebooted, it cannot be used and any tasks running on the current instance node will fail.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Ranger for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45290 PolicySync Direct Memory Usage Exceeds the Threshold
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks the direct memory usage of the PolicySync service every 60 seconds. This alarm is generated when the direct memory usage of the PolicySync instance exceeds the threshold (90% of the maximum memory) for five consecutive times. This alarm is cleared when the PolicySync direct memory usage is less than or equal to the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45290 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Direct memory overflow may cause service breakdown.
+Possible Causes
The direct memory of the PolicySync process is overused or the direct memory is inappropriately allocated.
+Handling Procedure
Check the direct memory usage.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms > ALM-45290 PolicySync Direct Memory Usage Exceeds the Threshold. Check the location information of the alarm and view the host name of the instance for which the alarm is generated.
- On FusionInsight Manager, choose Cluster > Services > Ranger > Instance. Select the role corresponding to the host name of the instance for which the alarm is generated. Click the drop-down list in the upper right corner of the chart area and choose Customize > CPU and Memory > PolicySync Direct Memory Usage. Click OK.
- Check whether the direct memory used by the PolicySync reaches the threshold (90% of the maximum direct memory by default). +
- On FusionInsight Manager, choose Cluster > Services > Ranger > Instance > PolicySync. Click Instance Configuration and then All Configurations, and choose PolicySync > System. Set -XX:MaxDirectMemorySize in the GC_OPTS parameter to a larger value based on site requirements and save the configuration.+
If this alarm is generated, the direct memory configured for PolicySync cannot meet the direct memory required by the PolicySync process. You are advised to check the direct memory usage of PolicySync and change the value of -XX:MaxDirectMemorySize in GC_OPTS to the twice of the direct memory used by PolicySync. You can change the value based on the actual service scenario. Refer to 2 to view the TokenServer direct memory usage.
+ - Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.+
When the service is rebooted, it becomes unavailable and can disrupt business operations. When the instance is rebooted, it cannot be used and any tasks running on the current instance node will fail.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Ranger for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45291 PolicySync Non-Heap Memory Usage Exceeds the Threshold
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks the non-heap memory usage of the PolicySync service every 60 seconds. This alarm is generated when the non-heap memory usage of the PolicySync instance exceeds the threshold (90% of the maximum memory) for five consecutive times. This alarm is cleared when the non-heap memory usage is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45291 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Non-heap memory overflow may cause service breakdown.
+Possible Causes
The non-heap memory of the PolicySync instance is overused or the non-heap memory is inappropriately allocated.
+Handling Procedure
Check non-heap memory usage.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms > ALM-45291 PolicySync Non-Heap Memory Usage Exceeds the Threshold. Check the location information of the alarm and view the host name of the instance for which the alarm is generated.
- On FusionInsight Manager, choose Cluster > Services > Ranger > Instance. Select the role corresponding to the host name of the instance for which the alarm is generated. Click the drop-down list in the upper right corner of the chart area and choose Customize > CPU and Memory > PolicySync Non-Heap Memory Usage. Click OK.
- Check whether the non-heap memory used by PolicySync reaches the threshold (90% of the maximum heap memory by default). +
- On FusionInsight Manager, choose Cluster > Services > Ranger > Instance > PolicySync. Click Instance Configuration and then All Configurations, and choose PolicySync > System. Set -XX: MaxPermSize in the GC_OPTS parameter to a larger value based on site requirements and save the configuration.+
If this alarm is generated, the non-heap memory size configured for the PolicySync instance cannot meet the non-heap memory required by the PolicySync process. You are advised to change the value of -XX:MaxPermSize in GC_OPTS to twice that of the current non-heap memory size or change the value based on site requirements.
+ - Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.+
When the service is rebooted, it becomes unavailable and can disrupt business operations. When the instance is rebooted, it cannot be used and any tasks running on the current instance node will fail.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Ranger for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45292 PolicySync GC Duration Exceeds the Threshold
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks the GC duration of the PolicySync process every 60 seconds. This alarm is generated when the GC duration of the PolicySync process exceeds the threshold for five consecutive times. This alarm is cleared when the GC duration is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45292 + |
+Critical (default threshold: 20000ms) +Major (default threshold: 12000ms) + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
PolicySync responds slowly.
+Possible Causes
The heap memory of the PolicySync process is overused or inappropriately allocated, causing frequent occurrence of the GC process.
+Handling Procedure
Check the GC duration.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms > ALM-45292 PolicySync GC Duration Exceeds the Threshold. Check the location information of the alarm and view the host name of the instance for which the alarm is generated.
- On FusionInsight Manager, choose Cluster > Services > Ranger > Instance. Select the role corresponding to the host name of the instance for which the alarm is generated and click the drop-down list in the upper right corner of the chart area. Choose Customize > GC > PolicySync GC Duration. Click OK.
- Check whether the GC duration of the PolicySync process collected every minute exceeds the threshold (12 seconds by default). +
- On FusionInsight Manager, choose Cluster > Services > Ranger > Instance > PolicySync. Click Instance Configuration and then All Configurations, and choose PolicySync > System. Set -Xmx in the GC_OPTS parameter to a larger value based on site requirements and save the configuration.+
If this alarm is generated, the heap memory configured for PolicySync cannot meet the heap memory required by the PolicySync process. You are advised to change the value of -Xmx in GC_OPTS to twice that of the heap memory used by PolicySync. You can change the value based on the actual service scenario. Refer to 2 to view the PolicySync heap memory usage.
+ - Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.+
When the service is rebooted, it becomes unavailable and can disrupt business operations. When the instance is rebooted, it cannot be used and any tasks running on the current instance node will fail.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Ranger for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45435 Inconsistent Metadata of ClickHouse Tables
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
This alarm is generated when the metadata in a distributed table or in the local table of the distributed table has been inconsistent for 180 min.
+This alarm is automatically cleared when the metadata in the distributed table or in the local table of the distributed table becomes consistent.
+Metadata consistency includes:
+- Consistent quantity, name, sequence, and type of each column in the table
- Consistent partition keys
- Consistent sorting keys
- Consistent primary keys
- Consistent sampling keys
If this alarm exists, table metadata is inconsistent in the ClickHouse cluster to which the current node belongs. The inconsistency may be caused by multiple reasons, not limited to those mentioned in additional information.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45435 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
Table + |
+Specifies the database name and table name for which the alarm is generated. + |
+
Impact on the System
Subsequent operations such as INSERT and ALTER on the table may fail.
+Possible Causes
Table metadata modification fails or is not executed on one or more ClickHouseServer nodes.
+Handling Procedure
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Log in to the node where the client is installed as the client installation user and run the following commands:
cd {Client installation path}
+source bigdata_env
+- For a cluster with Kerberos authentication enabled (security mode):
+
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 9440 --secure
+ - For a cluster with Kerberos authentication disabled (normal mode):
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user Username --password --port 9000
+
- For a cluster with Kerberos authentication enabled (security mode):
+
- Check whether any task is being executed for the table to which the alarm is generated.
Run the following command to check whether any SQL task is being executed:
+select * from system.processes where current_database='Database name' and query like '%Table name%'
+Run the following command to check whether a mutation task is being executed:
+select * from system.mutations where database='Database name' and table='Table name';
+- If the query result is empty, go to 4.
- If the query result contains error information, rectify the fault accordingly. If the fault cannot be rectified based on the error information, go to 6.
- If the query result contains information about an on-going task with no error, the SQL/mutation task is being executed.
Wait for 5 minutes. If the alarm is cleared, no further action is required. If the alarm persists, go to 4.
+
- Modify the table structure, delete a table, or add a table based on service requirements until the table metadata of all nodes in the cluster is consistent. After 5 minutes, check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
- If the table has been deleted, manually clear the alarm and check whether the alarm is reported again.
- If yes, go to 6.
- If no, no further action is required.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45436 Skew ClickHouse Table Data
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
This alarm is generated when data skew occurs in the local table of a distributed table between ClickHouse nodes. This alarm is automatically cleared when data becomes balanced.
+Data skew check method:
+- If min_table_check_data_bytes is set to 0, data skew check is disabled.
- If min_table_check_data_bytes is greater than 0, data skew check is enabled.
After data skew check is enabled, if the data volume in a table is less than the min_table_check_data_bytes value, no alarm will be reported due to data skew. When the data volume is greater than the min_table_check_data_bytes value and the data volume difference between the same table on different nodes is greater than the percentage specified in min_table_data_varies_rate, data skew occurs and this alarm is reported.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45436 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
Table + |
+Specifies the database name and table name for which the alarm is generated. + |
+
Impact on the System
SQL execution efficiency may be lowered.
+Possible Causes
The data write policy is improper, causing unbalanced data among nodes.
+Handling Procedure
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Log in to the node where the client is installed as the client installation user and run the following commands:
cd {Client installation path}
+source bigdata_env
+- Security mode (with Kerberos enabled):
+
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 9440 --secure
+ - Normal mode (with Kerberos disabled):
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user Username --password --port 9000
+
- Security mode (with Kerberos enabled):
+
- View data distribution.
select FQDN(), database, table, sum(data_compressed_bytes) from clusterAllReplicas(Name of the logical cluster, system.parts) where database='Database name' and table='Table name' and active=1 group by (FQDN(), database, table);
+ - Balance data with a few clicks or migrate data based on service requirements.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45437 Excessive Parts in the ClickHouse Table
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
This alarm is generated when the number of parts exceeds the threshold specified by part_num_threshold.
+This alarm is automatically cleared when the number of parts is less than the part_num_threshold value.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45437 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
Table + |
+Specifies the database name and table name for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
Service errors may occur.
+Possible Causes
The data distribution in the ClickHouse table is improper, or the background merge task is executed slowly.
+Handling Procedure
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Log in to the node where the client is installed as the client installation user and run the following commands:
cd {Client installation path}
+source bigdata_env
+- Security mode (with Kerberos enabled):
+
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 9440 --secure
+ - Normal mode (with Kerberos disabled):
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user Username --password --port 9000
+
- Security mode (with Kerberos enabled):
+
- Run the following command to manually merge parts:
optimize table Database name.Table name final;
+ - Check whether the number of parts has decreased.
select FQDN(), database, table, count(1) from clusterAllReplicas(default_cluster, system.parts) where database='Database name' and table='Table name' and active=1 group by (FQDN(), database, table);
+- If the number of parts is less than the threshold, wait for 5 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
- If the number of parts does not decrease, check whether the partition key of the table is set properly. If the number of partitions is too large, rectify the service logic.
- If the command output is empty, the table does not exist. This alarm is a historical alarm and can be ignored. Manually clear it.
- If the number of parts is less than the threshold, wait for 5 minutes and check whether the alarm is cleared.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45438 ClickHouse Disk Usage Exceeds 80%
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks the disk capacity of the ClickHouseServer node every 1 minute. This alarm is generated when the usage of the disk where the ClickHouse data directory or metadata directory resides exceeds 80%.
+This alarm is automatically cleared when the usage of the disk where the ClickHouse data directory or metadata directory is located is lower than 80%.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45438 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
DiskPath + |
+Specifies the path of the disk for which the alarm is generated. + |
+
Impact on the System
The ClickHouse write operation may fail.
+Possible Causes
The disk capacity of the ClickHouseServer node is too small.
+Handling Procedure
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Expand the disk capacity of the node for which the alarm is generated.
- Go to 4 if the expansion fails or the alarm persists after the expansion.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45439 ClickHouse Node Enters the Read-Only Mode
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks the disk capacity of the ClickHouseServer node every 1 minute. This alarm is generated when the system detects that the disk capacity exceeds 90% and the ClickHouseServer node enters the read-only mode.
+This alarm is automatically cleared when the system detects that the disk capacity is lower than 90% and the ClickHouseServer node exits the read-only mode.
+ If the ClickHouseServer node is in read-only mode and you need to log in to the client to clear data, you can manually exit the read-only mode using the following method:
+Log in to FusionInsight Manager, choose Cluster > Services > ClickHouse > Configurations > All Configurations, search for profiles.default.readonly, and change its value to 0.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45439 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
DiskPath + |
+Specifies the path of the disk for which the alarm is generated. + |
+
Impact on the System
After the ClickHouseServer node enters the read-only mode, all write, modification, and deletion operations fail.
+Possible Causes
The disk usage of the ClickHouse node exceeds 90%.
+Handling Procedure
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Expand the disk capacity of the node for which the alarm is generated.
- Go to 4 if the expansion fails or the alarm persists after the expansion.+
After the capacity expansion, this alarm can be automatically cleared only when profiles.default.readonly is auto. If its value has been manually changed, change it back to auto. If profiles.default.readonly needs to be set to 0 or 1 based on service requirements, manually clear this alarm.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45440 Inconsistency Between ClickHouse Replicas
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
When the number of ClickHouse replicas is greater than 1, the system periodically checks the replicated table. This alarm is generated if replicated table data is not synchronized. This alarm is cleared when data in all replicated tables between replicas becomes synchronized.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45440 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Table + |
+Specifies the table name for which the alarm is generated. + |
+
Impact on the System
The data reliability of the ClickHouse replicated table is affected, causing data differences and affecting the query result of the distributed table.
+Possible Causes
- The ClickHouse service is overloaded.
- The connection between the ClickHouse and ZooKeeper is abnormal.
Handling Procedure
Check whether the ClickHouse service load is heavy.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the database name, table name, role name and IP address for the hostname in Location.
- Log in to the node where the client is installed as the client installation user and run the following commands:
cd {Client installation path}
+source bigdata_env
+- For a cluster with Kerberos authentication enabled (security mode):
+
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 9440 --secure
+ - For a cluster with Kerberos authentication disabled (normal mode):
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user Username --password --port 9000
+
- For a cluster with Kerberos authentication enabled (security mode):
+
- Run the following statement to check whether data is frequently written to the system table. If yes, wait until the service execution is complete and check whether the alarm is cleared.
SELECT query_id, user, FQDN(), elapsed, query FROM system.processes ORDER BY query_id;
+- If yes, no further action is required.
- If no, go to 4.
- Check whether a large amount of data is written. If yes, wait until the task is complete and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
- Run the following statement to check whether replicas are synchronized:
select table,absolute_delay, queue_size, inserts_in_queue, merges_in_queue from system.replicas where absolute_delay > 0 order by absolute_delay desc limit 10;
+ + - If inserts_in_queue contains a large amount of content to be inserted, run the following SQL statement to query the replica synchronization queue and locate the error cause:
SELECT database,table,type,any(last_exception),any(postpone_reason),min(create_time),max(last_attempt_time),max(last_postpone_time),max(num_postponed) AS max_postponed,max(num_tries) AS max_tries,min(num_tries) AS min_tries,countIf(last_exception != '') AS count_err,countIf(num_postponed > 0) AS count_postponed,countIf(is_currently_executing) AS count_executing,count() AS count_all FROM system.replication_queue GROUP BY database,table,type ORDER BY count_all DESC
+Check whether an error message similar to the following is displayed:
+Not executing fetch of part xxx because n fetches already executing, max n
+ + - On FusionInsight Manager, choose Cluster > Services > ClickHouse > Configurations > All Configurations, and check whether the value of background_pool_size is twice the number of cores on the node. +
- Set this parameter to twice the number of cores on the node and synchronize the configuration. Wait for a while and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Check the connectivity between ClickHouse and ZooKeeper.
+- Log in to the node where the ClickHouseServer instance is located, go to ${BIGDATA_HOME}/FusionInsight_ClickHouse_*/*_*_ClickHouseServer/etc, and check whether the port numbers of the ClickHouseServer and ZooKeeper in the config.xml file are the same, as shown in the following information in bold:+
To view the ZooKeeper port number, choose Cluster > Services > ZooKeeper > Configurations > All Configurations on FusionInsight Manager, and check the value of clientPort.
+<zookeeper> + <session_timeout_ms>10000</session_timeout_ms> + <node index="1"> + <host>server-2110082001-0019</host> + <port>24002</port> + </node> + <node index="2"> + <host>server-2110082001-0018</host> + <port>24002</port> + </node> + <node index="3"> + <host>server-2110082001-0017</host> + <port>24002</port> + </node> + </zookeeper>
+ + - Change the port number to the ZooKeeper port number, restart the ClickHouseServer instance, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45441 Zookeeper Disconnected
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks the connection between ClickHouse and ZooKeeper every minute. This alarm is generated when the connection fails. The alarm is reported because the ZooKeeper connection is abnormal. If the connection fails for three consecutive times, the system generates an alarm.
+This alarm is automatically cleared when the system detects that the connection is normal.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45441 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
If ClickHouse is disconnected from ZooKeeper, the ClickHouse service cannot be used.
+Possible Causes
- The ZooKeeper service is abnormal.
- The ClickHouse service is overloaded.
Handling Procedure
Check whether ZooKeeper is normal.
+- On FusionInsight Manager, choose Cluster > Services > ZooKeeper > quorumpeer.
- Check whether ZooKeeper instances are normal. +
- Select instances whose status is not good and choose More > Restart Instance.
- Check whether the instance status is good after restart. +
- Choose O&M > Alarm > Alarms and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Check whether the ClickHouse service load is heavy.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Log in to the node where the client is installed as the client installation user and run the following commands:
cd {Client installation path}
+source bigdata_env
+- For a cluster with Kerberos authentication enabled (security mode):
+
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 9440 --secure
+ - For a cluster with Kerberos authentication disabled (normal mode):
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user User name --password --port 9440
+
- For a cluster with Kerberos authentication enabled (security mode):
+
- Run the following statement to check whether data is frequently written to the system table. If yes, wait until the service execution is complete and check whether the alarm is cleared.
SELECT query_id, user, FQDN(), elapsed, query FROM system.processes ORDER BY query_id;
+- If yes, no further action is required.
- If no, go to 9.
- Check whether a large amount of data is written. If yes, wait until the task is complete and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 10.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45442 Too Many Concurrent SQL Statements
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The alarm module checks the number of concurrent ClickHouse requests every 30 seconds. This alarm is generated when the number of concurrent ClickHouse requests exceeds the concurrency threshold configured on the UI.
+This alarm is cleared when the system detects that the actual number of concurrent requests is less than concurrency threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45442 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
If there are too many concurrent SQL statements, a large number of system resources are consumed. As a result, system response becomes slow.
+Possible Causes
The ClickHouse service is overloaded.
+Handling Procedure
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Choose Cluster > ClickHouse > Instance, select an instance based on the alarm information. Choose Chart > Concurrency to check whether the actual number of concurrent SQL statements is greater than SQL concurrency threshold. +
- Confirm with the user whether a large number of tasks were being executed during the alarming period. +
- On FusionInsight Manager, choose O&M and click Alarm > Thresholds in the navigation pane on the left. On the displayed page, click ClickHouse > Concurrency and adjust the threshold, or wait until the task is complete. Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45443 Slow SQL Queries in the Cluster
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks slow SQL queries for ClickHouse every 1 minute. This alarm is generated when the execution time of a SQL statement is longer than or equal to the slow SQL threshold.
+This alarm is automatically cleared when the system detects that the execution time of the SQL statement is shorter than the slow SQL threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45443 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
The performance of the ClickHouse service deteriorates, which slows the response of other services. If there are too many slow SQL statements, the service may be unavailable.
+Possible Causes
- The ClickHouse service is overloaded.
- The execution of SQL statements takes a long time.
Handling Procedure
Check whether the ClickHouse service load is heavy.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Log in to the node where the client is installed as the client installation user and run the following commands:
cd {Client installation path}
+source bigdata_env
+- For a cluster with Kerberos authentication enabled (security mode):
+
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port --secure
+ - For a cluster with Kerberos authentication disabled (normal mode):
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user Username --password --port
+
- For a cluster with Kerberos authentication enabled (security mode):
+
- Run the following statement to check whether data is frequently written to the system table. If yes, wait until the service execution is complete and check whether the alarm is cleared.
SELECT query_id, user, FQDN(), elapsed, query FROM system.processes ORDER BY query_id;
+- If yes, no further action is required.
- If no, go to 4.
Checking whether the SQL statements take a long time.
+- Check the logical cluster to which the alarm object belongs. Log in to FusionInsight Manager, click Cluster, choose Services > ClickHouse, and click Logic Cluster. On the displayed page, choose Query Management > Ongoing Slow Queries. Check which SQL statements take a long time on the displayed page, confirm with the user to adjust services, optimize slow SQL statements, and check whether the optimization is successful. +
- After the SQL statements are complete, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45444 Abnormal ClickHouse Process
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The health check module checks ClickHouse instances every 30 seconds. If the number of consecutive failures exceeds the threshold, an alarm is reported. In this case, the ClickHouse process may stop responding and services cannot be properly executed.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45444 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
If the ClickHouse process is abnormal, services cannot run properly.
+Possible Causes
The ClickHouse process runs improperly.
+Handling Procedure
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Log in to the node where the client is installed as the client installation user and run the following commands:
cd {Client installation path}
+source bigdata_env
+- For a cluster with Kerberos authentication enabled (security mode):
+
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 9440 --secure
+ - For a cluster with Kerberos authentication disabled (normal mode):
clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user Username --password --port 9000
+
- For a cluster with Kerberos authentication enabled (security mode):
+
- Run the following statement to check whether the result can be properly returned:
SELECT 1;
+ + - Wait for several minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select ClickHouse for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45639 Checkpointing of a Flink Job Times Out
+Description
The system checks the checkpointing timeout of Flink jobs every 30 seconds. This alarm is generated if the checkpointing timeout of a Flink job is longer than the threshold (600 seconds by default). This alarm is cleared when the checkpointing timeout of a job is less than or equal to the threshold.
+Attribute
+Alarm ID + |
+Alarm Severity + |
+Auto Clear + |
+
|---|---|---|
45639 + |
+Minor + |
+Yes + |
+
Parameters
+Name + |
+Meaning + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
UserName + |
+Specifies the username for which the alarm is generated. + |
+
Impact on the System
This alarm has no impact on the system.
+Possible Causes
The job may be in the sub-healthy state. The possible causes are as follows:
+- The memory for the TaskManager of the job is insufficient.
- The state memory is too large, making checkpointing time-consuming.
Procedure
- Log in to Manager as a user who has the FlinkServer management permission.
- Choose O&M > Alarm > Alarms > ALM-45639 Checkpointing of a Flink Job Times Out, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the failed task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view TaskManager logs.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the FlinkServer management permission. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
- View the logs of the failed job to rectify the fault, or contact the O&M personnel and send the collected fault logs. No further action is required.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/logs/Application ID of the failed job directory.
- View the logs of the failed job to rectify the fault, or contact the O&M personnel and send the collected fault logs.
Alarm Clearing
This alarm is cleared when the checkpointing timeout a Flink job is less than or equal to the threshold.
+Related Information
None
+ALM-45642 RocksDB Continuously Triggers Write Traffic Limiting
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration, 180s by default). This alarm is generated when RocksDB for a job continuously triggers write traffic limiting, that is, the RocksDB write rate is not 0. This alarm is cleared when the RocksDB write rate of the job becomes 0.
+The rocksdb.actual-delayed-write-rate parameter specifies the RocksDB write rate of a job. Value 0 indicates that the rate is not limited, and other values indicate traffic limiting.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45642 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
ApplicationName + |
+Specifies the name of the application for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
This alarm has no adverse impact on the system.
+Possible Causes
When the rate at which Flink jobs write data to RocksDB is not 0, write traffic limiting is triggered. The possible causes are as follows:
+- There are too many MemTables. As a result, write traffic is limited or write stops, and ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold is generated.
- The size of SST files at level 0 is too large, and ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold is generated.
- The estimated compaction size exceeds the threshold, and ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold is generated.
Handling Procedure
Check whether write traffic limiting or write stop is caused due to too many MemTables.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold.
- After ALM-45643 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check whether write traffic limiting or write stop is caused due to too many SST files at level 0.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold.
- After ALM-45644 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Check whether write traffic limiting or write stop is caused because the estimated compaction size exceeds the threshold.
+- In the alarm list, check whether ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold.
- After ALM-45647 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 12.
Collect fault information.
+- Log in to FusionInsight Manager as a user who has the FlinkServer management permission.
- Choose O&M > Alarm > Alarms > ALM-45642 RocksDB Continuously Triggers Write Traffic Limiting, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the abnormal task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the native Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view and save TaskManager logs.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the FlinkServer management permission. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
- View the job logs to rectify the fault, or contact the O&M personnel and send the collected fault logs. No further action is required.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/bucket-logs-tfile/Last four digits of the task application ID/Application ID of the task directory.
- View the logs of the failed job to rectify the fault, or contact the O&M personnel and send the collected fault logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration, 180s by default). This alarm is generated when the MemTable size of RocksDB for a job continuously exceeds the threshold (metrics.reporter.alarm.job.alarm.rocksdb.get.micros.threshold, 50000 microseconds by default). This alarm is cleared when the MemTable size of RocksDB for the job is less than or equal to the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45643 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
ApplicationName + |
+Specifies the name of the application for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
This alarm has no adverse impact on the system.
+Possible Causes
The write pressure of RocksDB is high.
+Handling Procedure
Check TaskManager logs for the write pressure of RocksDB and collect logs.
+- Log in to FusionInsight Manager as a user who has the FlinkServer management permission.
- Choose O&M > Alarm > Alarms > ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the abnormal task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view and save TaskManager logs. Then go to 7.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the management permission for the current Flink job. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/bucket-logs-tfile/Last four digits of the task application ID/Application ID of the task directory.
Check whether the write pressure of RocksDB is high.
+- Check whether the value of rocksdb.size-all-mem-tables (unit: byte) in the TaskManager monitoring logs (keyword RocksDBMetricPrint) is greater than or equal to the total write buffer size (Total write buffer = write_buffer_size x max_write_buffer_number).
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 8.
++
+Table 1 Custom parameters Parameter
+Default Value
+Description
+state.backend.rocksdb.writebuffer.count
+- 2
- 4: enables SPINNING_DISK_OPTIMIZED_HIGH_MEM.
- Number of buffers
- 2 to 10 are recommended. Adjust the value based on service requirements.
state.backend.rocksdb.writebuffer.size
+64MB
+- Buffer size
- 64MB to 256MB are recommended.
state.backend.rocksdb.thread.num
+- 2
- 4: enables SPINNING_DISK_OPTIMIZED_HIGH_MEM.
- Number of flush threads. Increase the number of threads to quickly flush memory data to disks.
- When the number of threads is increased, the number of vCores also needs to be increased.
- 2 to 10 are recommended.
- If no, go to 9.
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 8.
+
- Restart the job and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
- Contact O&M personnel and send the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration, 180s by default). This alarm is generated when the number of SST files at level 0 of RocksDB for a job continuously exceeds the threshold (state.backend.rocksdb.level0_slowdown_writes_trigger, 20 by default). This alarm is cleared when the number of SST files at level 0 of RocksDB for the job is less than or equal to the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45644 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
ApplicationName + |
+Specifies the name of the application for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
This alarm has no adverse impact on the system.
+Possible Causes
Possible causes are as follows:
+- The compaction pressure of RocksDB is too high, and ALM-45646 Pending Compaction Size of RocksDB Continuously Exceeds the Threshold and ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold are generated.
- There are too many SST files at level 0.
Handling Procedure
Check whether the compaction pressure of RocksDB is too high and ALM-45646 is generated.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether ALM-45646 Pending Compaction Size of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45646 Pending Compaction Size of RocksDB Continuously Exceeds the Threshold.
- After ALM-45646 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check whether the compaction pressure of RocksDB is too high and ALM-45647 is generated.
+- In the alarm list, check whether ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold.
- After ALM-45647 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Check TaskManager logs for the number of SST files at level 0 and collect logs.
+- Log in to FusionInsight Manager as a user who has the FlinkServer management permission.
- Choose O&M > Alarm > Alarms > ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the abnormal task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view and save TaskManager logs. Then go to 14.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the management permission for the current Flink job. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/bucket-logs-tfile/Last four digits of the task application ID/Application ID of the task directory.
Check whether the number of SST files at level 0 is too large.
+- Check whether the value of rocksdb.num-files-at-level0 in TaskManager monitoring logs (keyword RocksDBMetricPrint) is greater than or equal to the value of state.backend.rocksdb.level0_slowdown_writes_trigger or state.backend.rocksdb.level0_stop_writes_trigger.
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 15.
++
+Table 1 Custom parameters Parameter
+Default Value
+Description
+state.backend.rocksdb.level0_slowdown_writes_trigger
+20
+- Number of files that trigger slowdown at level 0
- 20 to 30 are recommended.
state.backend.rocksdb.level0_stop_writes_trigger
+36
+- Maximum number of files that trigger stop at level 0
- 36 to 46 are recommended.
- If no, go to 16.
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 15.
+
- Restart the job and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 16.
- Contact O&M personnel and send the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45645 Pending Flush Size of RocksDB Continuously Exceeds the Threshold
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration, 180s by default). This alarm is generated when the number of pending flush requests of RocksDB for a job continuously reaches n times the number of flush/compaction threads. This alarm is cleared when the number of pending flush requests of RocksDB for the job is less than or equal to the threshold.
+- The number of flush/compaction threads is the value of state.backend.rocksdb.thread.num. The default value is 2. If SPINNING_DISK_OPTIMIZED_HIGH_MEM is enabled, the default value is 4.
- The metrics.reporter.alarm.job.alarm.rocksdb.background.jobs.multiplier parameter specifies n times the number of flush/compaction threads. The default value is 2.
Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45645 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
ApplicationName + |
+Specifies the name of the application for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
This alarm has no adverse impact on the system.
+Possible Causes
The number of pending flush requests of RocksDB for the Flink job is too large.
+Handling Procedure
Check TaskManager logs for the number of pending flush requests and collect logs.
+- Log in to FusionInsight Manager as a user who has the FlinkServer management permission.
- Choose O&M > Alarm > Alarms > ALM-45645 Pending Flush Size of RocksDB Continuously Exceeds the Threshold, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the abnormal task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view and save TaskManager logs. Then go to 7.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the management permission for the current Flink job. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/bucket-logs-tfile/Last four digits of the task application ID/Application ID of the task directory.
Check whether there are too many pending flush requests.
+- Check whether the sum of the values of rocksdb.mem-table-flush-pending and rocksdb.compaction-pending in TaskManager monitoring logs (keyword RocksDBMetricPrint) is greater than n times the number of RocksDB threads (metrics.reporter.alarm.job.alarm.rocksdb.background.jobs.multiplier, 2 by default). If it is, you can increase the number of RocksDB threads.
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 8.
++
+Table 1 Custom parameters Parameter
+Default Value
+Description
+state.backend.rocksdb.thread.num
+- 2
- 4: enables SPINNING_DISK_OPTIMIZED_HIGH_MEM.
- Number of flush threads. Increase the number of threads to quickly flush memory data to disks.
- When the number of threads is increased, the number of vCores also needs to be increased.
- 2 to 10 are recommended.
- If no, go to 9.
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 8.
+
- Restart the job and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
- Contact O&M personnel and send the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45646 Pending Compaction Size of RocksDB Continuously Exceeds the Threshold
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration, 180s by default). This alarm is generated when the number of pending compaction requests of RocksDB for a job continuously reaches n times the number of flush/compaction threads. This alarm is cleared when the number of pending compaction requests of RocksDB for the job is less than or equal to the threshold.
+- The number of flush/compaction threads is the value of state.backend.rocksdb.thread.num. The default value is 2. If SPINNING_DISK_OPTIMIZED_HIGH_MEM is enabled, the default value is 4.
- The metrics.reporter.alarm.job.alarm.rocksdb.background.jobs.multiplier parameter specifies n times the number of flush/compaction threads. The default value is 2.
Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45646 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
ApplicationName + |
+Specifies the name of the application for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
This alarm has no adverse impact on the system.
+Possible Causes
The number of pending compaction requests of RocksDB for the Flink job is too large.
+Handling Procedure
Check TaskManager logs for the number of pending compaction requests and collect logs.
+- Log in to FusionInsight Manager as a user who has the FlinkServer management permission.
- Choose O&M > Alarm > Alarms > ALM-45646 Pending Compaction Size of RocksDB Continuously Exceeds the Threshold, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the abnormal task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view and save TaskManager logs. Then go to 7.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the management permission for the current Flink job. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/bucket-logs-tfile/Last four digits of the task application ID/Application ID of the task directory.
Check whether there are too many pending compaction requests.
+- Check whether the sum of the values of rocksdb.mem-table-flush-pending and rocksdb.compaction-pending in TaskManager monitoring logs (keyword RocksDBMetricPrint) is greater than n times the number of RocksDB threads (metrics.reporter.alarm.job.alarm.rocksdb.background.jobs.multiplier, 2 by default). If it is, you can increase the number of RocksDB threads.
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 8.
++
+Table 1 Custom parameters Parameter
+Default Value
+Description
+state.backend.rocksdb.thread.num
+- 2
- 4: enables SPINNING_DISK_OPTIMIZED_HIGH_MEM.
- Number of flush threads. Increase the number of threads to quickly flush memory data to disks.
- When the number of threads is increased, the number of vCores also needs to be increased.
- 2 to 10 are recommended.
- If no, go to 9.
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 8.
+
- Restart the job and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration, 180s by default). This alarm is generated when the estimated pending compaction size of RocksDB for a job continuously exceeds the threshold. This alarm is cleared when the estimated pending compaction size of RocksDB for the job is less than or equal to the threshold.
+The threshold of the estimated pending compaction size is the smaller value of the following two parameters:
+- state.backend.rocksdb.soft-pending-compaction-bytes-limit. The default value is 64GB.
- state.backend.rocksdb.hard-pending-compaction-bytes-limit. The default value is 256GB.
Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45647 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
ApplicationName + |
+Specifies the name of the application for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
This alarm has no adverse impact on the system.
+Possible Causes
The estimated compaction data size of RocksDB is too large.
+Handling Procedure
Check TaskManager logs for the estimated compaction data size and collect logs.
+- Log in to FusionInsight Manager as a user who has the FlinkServer management permission.
- Choose O&M > Alarm > Alarms > ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the abnormal task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view and save TaskManager logs. Then go to 7.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the management permission for the current Flink job. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/bucket-logs-tfile/Last four digits of the task application ID/Application ID of the task directory.
Check whether the estimated compaction data size of RocksDB is too large.
+- Check whether the value of rocksdb.estimate-pending-compaction-bytes (unit: byte) in TaskManager monitoring logs (keyword RocksDBMetricPrint) is greater than or equal to the soft/hard-pending-compaction size (values of state.backend.rocksdb.soft-pending-compaction-bytes-limit and state.backend.rocksdb.hard-pending-compaction-bytes-limit).
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 8.
++
+Table 1 Custom parameters Parameter
+Default Value
+Description
+state.backend.rocksdb.soft-pending-compaction-bytes-limit
+64GB
+- When the pending compaction size exceeds the threshold, the write traffic is limited.
- 64GB to 512GB are recommended.
state.backend.rocksdb.hard-pending-compaction-bytes-limit
+256GB
+- When the pending compaction size exceeds the threshold, write operations are stopped.
- 64GB to 512GB are recommended.
- If no, go to 9.
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 8.
+
- Restart the job and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
- Contact O&M personnel and send the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45648 RocksDB Frequently Encounters Write-Stopped
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration, 180s by default). This alarm is generated when RocksDB for a job continuously encounters the is-write-stopped state. This alarm is cleared when RocksDB for the job no longer or does not continuously encounter the is-write-stopped state within an alarm reporting interval.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45648 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
ApplicationName + |
+Specifies the name of the application for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
This alarm has no adverse impact on the system.
+Possible Causes
The possible causes are as follows:
+- There are too many MemTables and ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold is generated.
- There are too many SST files at level 0, and ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold is generated.
- The estimated compaction size exceeds the threshold, and ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold is generated.
Handling Procedure
Check whether there are too many MemTables.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold.
- After ALM-45643 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check whether the number of SST files at level 0 is too large.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold.
- After ALM-45644 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Check whether the estimated compaction size exceeds the threshold.
+- In the alarm list, check whether ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold.
- After ALM-45647 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 12.
Collect fault information.
+- Log in to Manager as a user who has the management permission for the current Flink job.
- Choose O&M > Alarm > Alarms > ALM-45648 RocksDB Frequently Encounters Write-Stopped, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the abnormal task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view and save TaskManager logs.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the management permission for the current Flink job. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
- View the job logs to rectify the fault, or contact the O&M personnel and send the collected fault logs. No further action is required.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/bucket-logs-tfile/Last four digits of the task application ID/Application ID of the task directory.
- View the logs of the failed job to rectify the fault, or contact the O&M personnel and send the collected fault logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45649 P95 Latency of RocksDB Get Requests Continuously Exceeds the Threshold
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration, 180s by default). This alarm is generated when the P95 latency of RocksDB Get requests exceeds the threshold (metrics.reporter.alarm.job.alarm.rocksdb.get.micros.threshold, 50000 microseconds by default). This alarm is cleared when the P95 latency of RocksDB Get requests is less than or equal to the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45649 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
ApplicationName + |
+Specifies the name of the application for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
This alarm has no adverse impact on the system.
+Possible Causes
The possible causes are as follows:
+- There are too many SST files at level 0, causing slow queries. In addition, ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold is generated.
- The cache hit ratio is lower than 60%, causing frequent swap-ins and swap-outs of the block cache.
Handling Procedure
Check whether the number of SST files at level 0 is too large.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold.
- After ALM-45644 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check the cache hit ratio in TaskManager logs and collect logs.
+- Log in to FusionInsight Manager as a user who has the FlinkServer management permission.
- Choose O&M > Alarm > Alarms > ALM-45649 P95 Latency of RocksDB Get Requests Continuously Exceeds the Threshold, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the abnormal task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view and save TaskManager logs. Then go to 11.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the management permission for the current Flink job. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/bucket-logs-tfile/Last four digits of the task application ID/Application ID of the task directory.
Check whether the cache hit ratio is too low.
+- Check the values of rocksdb.block.cache.hit (cache hit) and rocksdb.block.cache.miss (cache miss) in TaskManager monitoring logs (keyword RocksDBMetricPrint). Calculate the hit ratio using the following formula and check whether it is less than 60%:rocksdb.block.cache.hit/(rocksdb.block.cache.hit+rocksdb.block.cache.miss)+
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 12.
++
+Table 1 Custom parameters Parameter
+Default Value
+Description
+state.backend.rocksdb.block.cache-size
+- 8MB
- 256MB: enables SPINNING_DISK_OPTIMIZED_HIGH_MEM.
- Cache size
- 8MB to 1GB are recommended.
state.backend.rocksdb.block.blocksize
+- 4KB
- 128KB: enables SPINNING_DISK_OPTIMIZED_HIGH_MEM.
- Block size
- 4KB to 256KB are recommended.
state.backend.rocksdb.use-bloom-filter
+false
+- Whether to speed up indexing. If it is true, each new SST file will contain a Bloom filter.
- true is recommended.
- If no, go to 13.
- If yes, adjust the values of the following custom parameters on the job development page of the Flink web UI, save the settings, and go to 12.
+
- Restart the job and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 13.
- Contact O&M personnel and send the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45650 P95 Latency of RocksDB Write Requests Continuously Exceeds the Threshold
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The system checks the RocksDB monitoring data of jobs at the user-specified alarm reporting interval (metrics.reporter.alarm.job.alarm.rocksdb.metrics.duration, 180s by default). This alarm is generated when the P95 latency of RocksDB write requests exceeds the threshold (metrics.reporter.alarm.job.alarm.rocksdb.write.micros.threshold, 50000 microseconds by default). This alarm is cleared when the P95 latency of RocksDB write requests is less than or equal to the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45650 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
ApplicationName + |
+Specifies the name of the application for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
JobName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
This alarm has no adverse impact on the system.
+Possible Causes
The possible causes are as follows:
+- There are too many MemTables. As a result, write traffic is limited or write stops, and ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold is generated.
- There are too many SST files at level 0, and ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold is generated.
- The estimated compaction size exceeds the threshold, and ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold is generated.
Handling Procedure
Check whether write traffic limiting or write stop is caused due to too many MemTables.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold.
- After ALM-45643 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check whether the number of SST files at level 0 is too large.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold.
- After ALM-45644 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Check whether the estimated compaction size exceeds the threshold.
+- In the alarm list, check whether ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold exists. +
- Handle the alarm by following the instructions provided in section ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold.
- After ALM-45647 is cleared, wait a few minutes and check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 12.
Collect fault information.
+- Log in to FusionInsight Manager as a user who has the FlinkServer management permission.
- Choose O&M > Alarm > Alarms > ALM-45650 P95 Latency of RocksDB Write Requests Continuously Exceeds the Threshold, view Location, and obtain the name of the task for which the alarm is generated.
- Choose Cluster > Services > Yarn and click the link next to ResourceManager WebUI to go to the native Yarn page.
- Locate the abnormal task based on its name displayed in Location, search for and record the application ID of the job, and check whether the job logs are available on the Yarn page.Figure 1 Application ID of a job+ +
- Click the application ID of the failed job to go to the job page.
- Click Logs in the Logs column to view JobManager logs.Figure 2 Clicking Logs+
- Click the ID in the Attempt ID column and click Logs in the Logs column to view and save TaskManager logs.Figure 3 Clicking the ID in the Attempt ID column+
Figure 4 Clicking Logs+
+ You can also log in to Manager as a user who has the FlinkServer management permission. Choose Cluster > Services > Flink, and click the link next to Flink WebUI. On the displayed Flink web UI, click Job Management, click More in the Operation column, and select Job Monitoring to view TaskManager logs.
+
- Click Logs in the Logs column to view JobManager logs.
- View the job logs to rectify the fault, or contact the O&M personnel and send the collected fault logs. No further action is required.
If logs are unavailable on the Yarn page, download logs from HDFS.
+- On Manager, choose Cluster > Services > HDFS, click the link next to NameNode WebUI to go to the HDFS page, choose Utilities > Browse the file system, and download logs in the /tmp/logs/Username/bucket-logs-tfile/Last four digits of the task application ID/Application ID of the task directory.
- View the logs of the failed job to rectify the fault, or contact the O&M personnel and send the collected fault logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45652 Flink Service Unavailable
+This section applies to MRS 3.3.0 or later.
+Alarm Description
The alarm module checks the Flink status every 60 seconds. This alarm is generated when the Flink service is unavailable. This alarm is cleared when the Flink service recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45652 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the job for which the alarm is generated. + |
+
Impact on the System
FlinkServer and the Flink client cannot be used to submit Flink jobs.
+Possible Causes
The ZooKeeper, HDFS, Yarn, KrbServer, or DBService service on which Flink depends is unavailable.
+Handling Procedure
Check whether the ZooKeeper service on which Flink depends is abnormal.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms.
- In the alarm list, check whether "ALM-13000 ZooKeeper Service Unavailable" exists. +
- Handle the alarm by referring to "ALM-13000 ZooKeeper Service Unavailable."
- After the alarm is cleared, wait a few minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check whether the HDFS service on which Flink depends is abnormal.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether "ALM-14000 HDFS Service Unavailable" exists. +
- Handle the alarm by referring to "ALM-14000 HDFS Service Unavailable."
- After the alarm is cleared, wait a few minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Check whether the Yarn service on which Flink depends is abnormal.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether "ALM-18000 Yarn Service Unavailable" exists. +
- Handle the alarm by referring to "ALM-18000 Yarn Service Unavailable."
- After the alarm is cleared, wait a few minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 13.
Check whether the KrbServer service on which Flink depends is abnormal.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether "ALM-25500 KrbServer Service Unavailable" exists. +
- Handle the alarm by referring to "ALM-25500 KrbServer Service Unavailable."
- After the alarm is cleared, wait a few minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 17.
Check whether the DBService service on which Flink depends is abnormal.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms.
- In the alarm list, check whether "ALM-27001 DBService Service Unavailable" exists. +
- Handle the alarm by referring to "ALM-27001 DBService Service Unavailable."
- After the alarm is cleared, wait a few minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 21.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flink for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45653 Invalid Flink HA Certificate File
+This section applies to MRS 3.3.0 or later.
+Alarm Description
Flink checks whether the HA certificate file is valid (whether the certificate exists and whether its format is correct) in the first health check or at 01:00:00 every day. This alarm is generated when the certificate file is invalid. This alarm is automatically cleared when the certificate file becomes valid again.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45653 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
Some functions are unavailable.
+Possible Causes
The HA certificate file is invalid.
+Handling Procedure
View alarm information.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms > ALM-45653 Invalid Flink HA Certificate File, view Location, obtain the name of the host for which the alarm is generated, and click the host name to view its IP address.
Check whether the HA certificate file in the system is valid.
+- Log in to the host for which the alarm is generated as user omm.
- Run the cd ${BIGDATA_HOME}/FusionInsight_Flink_*/install/FusionInsight-Flink-*/ha/local/cert command to go to the directory where the HA certificate is stored.
- Run the ls -l command to check whether the server.crt file exists. +
- Run the openssl x509 -in server.crt -text -noout command and check whether the command output is normal. +
- Run the cd ${BIGDATA_HOME}/FusionInsight_Flink_*/install/FusionInsight-Flink-*/flink/sbin command to go to the Flink script directory.
- Run the sh proceed_ha_ssl_cert.sh command to generate a new HA certificate. Then, check whether the alarm is cleared 1 minute later. +
- Check whether this alarm is generated again during periodic system check.
- If yes, go to 9.
- If no, no further action is required.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flink for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45654 Flink HA Certificate Is About to Expire
+This section applies to MRS 3.3.0 or later.
+Alarm Description
Flink checks whether the HA certificate file is about to expire in the first health check or at 01:00:00 every day. This alarm is generated when the remaining validity period is less than or equal to 30 days. This alarm is automatically cleared when the remaining validity period is greater than 30 days.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45654 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
Currently, there is no impact on the system.
+Possible Causes
The HA certificate is about to expire.
+Handling Procedure
View alarm information.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms > ALM-45654 Flink HA Certificate Is About to Expire, view Location, obtain the name of the host for which the alarm is generated, and click the host name to view its IP address.
Check whether the HA certificate file in the system is valid. If it is not, generate a new one.
+- Log in to the host for which the alarm is generated as user omm.
- Run the cd ${BIGDATA_HOME}/FusionInsight_Flink_*/install/FusionInsight-Flink-*/ha/local/cert command to go to the directory where the HA certificate is stored.
- Run the openssl x509 -noout -text -in server.crt command to query the effective time and due time of the HA certificate.
- Perform 6 to 7 during off-peak hours to update the certificate file as needed.
- Run the cd ${BIGDATA_HOME}/FusionInsight_Flink_*/install/FusionInsight-Flink-*/flink/sbin command to go to the Flink script directory.
- Run the sh proceed_ha_ssl_cert.sh command to generate a new HA certificate. Then, check whether the alarm is cleared 1 minute later. +
- On the node where the standby FlinkServer instance is located, repeat 6 to 7. Then, check whether the alarm is cleared 1 minute later. +
- Check whether this alarm is generated again during periodic system check.
- If yes, go to 10.
- If no, no further action is required.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flink for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45655 Flink HA Certificate File Has Expired
+This section applies to MRS 3.3.0 or later.
+Alarm Description
Flink checks whether the HA certificate file has expired in the first health check or at 01:00:00 every day. This alarm is generated when the HA certificate has expired. This alarm is automatically cleared when the certificate file becomes valid again.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45655 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
Some functions are unavailable.
+Possible Causes
The HA certificate file has expired.
+Handling Procedure
View alarm information.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms > ALM-45655 Flink HA Certificate File Has Expired, view Location, obtain the name of the host for which the alarm is generated, and click the host name to view its IP address.
Check whether the HA certificate file in the system is valid. If it is not, generate a new one.
+- Log in to the host for which the alarm is generated as user omm.
- Run the cd ${BIGDATA_HOME}/FusionInsight_Flink_*/install/FusionInsight-Flink-*/ha/local/cert command to go to the directory where the HA certificate is stored.
- Run the openssl x509 -noout -text -in server.crt command to query the effective time and due time of the HA certificate and check whether the HA certificate file is valid. +
- Run the cd ${BIGDATA_HOME}/FusionInsight_Flink_*/install/FusionInsight-Flink-*/flink/sbin command to go to the Flink script directory.
- Run the sh proceed_ha_ssl_cert.sh command to generate a new HA certificate. Then, check whether the alarm is cleared 1 minute later. +
- On the node where the standby FlinkServer instance is located, repeat 5 to 6. Then, check whether the alarm is cleared 1 minute later. +
- Check whether this alarm is generated again during periodic system check.
- If yes, go to 9.
- If no, no further action is required.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Flink for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-45736 Guardian Service Unavailable
+Alarm Description
The alarm module checks the Guardian service status every 60 seconds. This alarm is generated if Guardian is unavailable.
+This alarm is cleared after Guardian recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45736 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+
RoleName + |
+Specifies the role for which the alarm was generated. + |
+
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Impact on the System
Guardian cannot work properly.
+Possible Causes
- The HDFS service on which the Guardian service depends is abnormal.
- The TokenServer role instance is abnormal.
Handling Procedure
Check the HDFS service status.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. On the page that is displayed, check whether "ALM-14000 HDFS Service Unavailable" is reported. +
- Clear this alarm according to the alarm help.After the alarm is cleared, wait a few minutes and check whether the alarm GuardianService Unavailable is cleared.+
- If yes, no further action is required.
- If no, go to 3.
Check all TokenServer instances.
+- Log in to the node where the TokenServer instance resides as user omm and run the ps -ef|grep "guardian.token.server.Server" command to check whether the TokenServer process exists on the node. +
- In the alarm list, check whether the alarm "Guardian Service Unavailable" is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the Service drop-down list, and select Guardian for the destination cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-45737 TokenServer Heap Memory Usage Exceeds the Threshold
+Alarm Description
The system checks the heap memory usage of the TokenServer service every 60 seconds. This alarm is generated when the heap memory usage of the TokenServer instance exceeds the threshold (95% of the maximum memory) for 10 consecutive times.
+This alarm is automatically cleared when the system detects that the heap memory usage is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45737 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+
RoleName + |
+Specifies the role for which the alarm was generated. + |
+
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Heap memory overflow may cause service unavailability.
+Possible Causes
The heap memory of the TokenServer instance is overused or the heap memory is inappropriately allocated.
+Handling Procedure
Check heap memory usage.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms > ALM-45737 TokenServer Heap Memory Usage Exceeds the Threshold. Check the location information of the alarm and view the host name of the instance for which the alarm is generated.
- On FusionInsight Manager, choose Cluster > Services > Guardian. On the page that is displayed, click the Instance tab. On this tab page, select the role corresponding to the host name of the instance for which the alarm is generated. Click the drop-down list in the upper right corner of the chart area and choose Customize > CPU and Memory > TokenServer Heap Memory Usage. Then click OK.
- Check whether the heap memory used by TokenServer reaches the threshold (95% of the maximum heap memory by default). +
- On FusionInsight Manager, choose Cluster > Services > Guardian. On the page that is displayed, click the Instance tab. On this tab page, choose TokenServer > Instance Configuration. Click All Configurations, and choose TokenServer > System. Set -Xmx in the GC_OPTS parameter to a larger value based on site requirements and save the configuration.+
If this alarm is generated, the heap memory configured for TokenServer cannot meet the heap memory required by the TokenServer process. You are advised to change the value of -Xmx in GC_OPTS to twice that of the heap memory used by TokenServer. You can change the value based on the actual service scenario. Refer to 2 to view the TokenServer heap memory usage.
+ - Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
+ During service or instance restart, Guardian may fail to be accessed and jobs cannot access OBS.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the Service drop-down list, and select Guardian for the destination cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-45738 TokenServer Direct Memory Usage Exceeds the Threshold
+Alarm Description
The system checks the direct memory usage of the TokenServer service every 60 seconds. This alarm is generated when the direct memory usage of the TokenServer instance exceeds the threshold (80% of the maximum memory) for five consecutive times.
+This alarm is automatically cleared when the system detects that the TokenServer direct memory usage is less than or equal to the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45738 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+
RoleName + |
+Specifies the role for which the alarm was generated. + |
+
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Direct memory overflow may cause service unavailability.
+Possible Causes
The direct memory of the TokenServer process is overused or the direct memory is inappropriately allocated.
+Handling Procedure
Check the direct memory usage.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms > ALM-45738 TokenServer Direct Memory Usage Exceeds the Threshold. Check the location information of the alarm and view the host name of the instance for which the alarm is generated.
- On FusionInsight Manager, choose Cluster > Services > Guardian. On the page that is displayed, click the Instance tab. On this tab page, select the role corresponding to the host name of the instance for which the alarm is generated. Click the drop-down list in the upper right corner of the chart area and choose Customize > CPU and Memory > TokenServer Direct Memory Usage. Then click OK.
- Check whether the direct memory used by TokenServer reaches the threshold (80% of the maximum direct memory by default). +
- On FusionInsight Manager, choose Cluster > Services > Guardian. On the page that is displayed, click the Instance tab. On this tab page, choose TokenServer > Instance Configuration. Click All Configurations, and choose TokenServer > System. Set -XX:MaxDirectMemorySize in the GC_OPTS parameter to a larger value based on site requirements and save the configuration.+
If this alarm is generated, the direct memory configured for TokenServer cannot meet the direct memory required by the TokenServer process. You are advised to check the direct memory usage of TokenServer and change the value of -XX:MaxDirectMemorySize in GC_OPTS to the twice of the direct memory used by TokenServer. You can change the value based on the actual service scenario. Refer to 2 to view the TokenServer direct memory usage.
+ - Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
+ During service or instance restart, Guardian may fail to be accessed and jobs cannot access OBS.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the Service drop-down list, and select Guardian for the destination cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-45739 TokenServer Non-Heap Memory Usage Exceeds the Threshold
+Alarm Description
The system checks the heap memory usage of the TokenServer service every 60 seconds. This alarm is generated when the non-heap memory usage of the TokenServer instance exceeds the threshold (80% of the maximum memory) for five consecutive times.
+This alarm is automatically cleared when the system detects that the non-heap memory usage is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45739 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+
RoleName + |
+Specifies the role for which the alarm was generated. + |
+
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Non-heap memory overflow may cause service unavailability.
+Possible Causes
The non-heap memory of the TokenServer instance is overused or the non-heap memory is inappropriately allocated.
+Handling Procedure
Check non-heap memory usage.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms > ALM-45739 TokenServer Non-Heap Memory Usage Exceeds the Threshold. Check the location information of the alarm and view the host name of the instance for which the alarm is generated.
- On FusionInsight Manager, choose Cluster > Services > Guardian. On the page that is displayed, click the Instance tab. On this tab page, select the role corresponding to the host name of the instance for which the alarm is generated. Click the drop-down list in the upper right corner of the chart area and choose Customize > CPU and Memory > TokenServer Non-Heap Memory Usage. Then click OK.
- Check whether the non-heap memory used by TokenServer reaches the threshold (80% of the maximum non-heap memory by default). +
- On FusionInsight Manager, choose Cluster > Services > Guardian. On the page that is displayed, click the Instance tab. On this tab page, choose TokenServer > Instance Configuration. Click All Configurations, and choose TokenServer > System. Set -XX: MaxPermSize in the GC_OPTS parameter to a larger value based on site requirements and save the configuration.+
If this alarm is generated, the non-heap memory size configured for the TokenServer instance cannot meet the non-heap memory required by the TokenServer process. You are advised to change the value of -XX:MaxPermSize in GC_OPTS to twice that of the current non-heap memory size or change the value based on site requirements.
+ - Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
+ During service or instance restart, Guardian may fail to be accessed and jobs cannot access OBS.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the Service drop-down list, and select Guardian for the destination cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-45740 TokenServer GC Duration Exceeds the Threshold
+Alarm Description
The system checks the GC duration of the TokenServer process every 60 seconds. This alarm is generated when the GC duration of the TokenServer process exceeds the threshold (12 seconds by default) for five consecutive times.
+This alarm is automatically cleared when the system detects that the GC duration is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45740 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+
RoleName + |
+Specifies the role for which the alarm was generated. + |
+
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
TokenServer responds slowly.
+Possible Causes
The heap memory of the TokenServer process is overused or inappropriately allocated, causing frequent occurrence of the GC process.
+Handling Procedure
Check the GC duration.
+- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms > ALM-45740 TokenServer GC Duration Exceeds the Threshold. Check the location information of the alarm and view the host name of the instance for which the alarm is generated.
- On FusionInsight Manager, choose Cluster > Services > Guardian. On the page that is displayed, click the Instance tab. On this tab page, select the role corresponding to the host name of the instance for which the alarm is generated and click the drop-down list in the upper right corner of the chart area. Choose Customize > GC > TokenServer GC Duration. Then click OK.
- Check whether the GC duration of the TokenServer process collected every minute exceeds the threshold (12 seconds by default). +
- On FusionInsight Manager, choose Cluster > Services > Guardian. On the page that is displayed, click the Instance tab. On this tab page, choose TokenServer > Instance Configuration. Click All Configurations, and choose TokenServer > System. Set -Xmx in the GC_OPTS parameter to a larger value based on site requirements and save the configuration.+
If this alarm is generated, the heap memory configured for TokenServer cannot meet the heap memory required by the TokenServer process. You are advised to change the value of -Xmx in GC_OPTS to twice that of the heap memory used by TokenServer. You can change the value based on the actual service scenario. Refer to 2 to view the TokenServer heap memory usage.
+ - Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.
+ During service or instance restart, Guardian may fail to be accessed and jobs cannot access OBS.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M > Log > Download.
- Expand the Service drop-down list, and select Guardian for the destination cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-45741 Failed to Call the ECS securitykey API
+Alarm Description
Guardian caches the temporary AK/SK of the ECS agency. When the cache does not exist or is about to expire, Guardian calls the securitykey API of ECS to update the AK/SK. This alarm is generated when calling to the API fails.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45741 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+
RoleName + |
+Specifies the role for which the alarm was generated. + |
+
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Impact on the System
The task may fail to obtain the temporary AK/SK for accessing OBS. As a result, the task cannot access OBS.
+Possible Causes
- No ECS agency is bound to the cluster.
- An underlying interface of ECS is abnormal.
Handling Procedure
Check whether an agency is bound to the cluster.
+- Log in to the MRS console.
- In the navigation pane on the left, choose Clusters > Active Clusters. On the page that is displayed, click the cluster name to go to its overview page. Then, check whether the cluster is bound to an agency in the O&M management area. +
- Click Manage Agency. On the page that is displayed, rebind the cluster to an agency. Then check whether the alarm is cleared a few minutes later.
- If yes, no further action is required.
- If no, go to 4.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Guardian for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-45742 Failed to Call the ECS Metadata API
+Alarm Description
When Guardian calls an IAM API to obtain the temporary AK/SK, it needs to first obtain related metadata via the ECS Metadata API. This alarm is generated when Guardian fails to call the Metadata API.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45742 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+
RoleName + |
+Specifies the role for which the alarm was generated. + |
+
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Impact on the System
The task may fail to obtain the temporary AK/SK for accessing OBS. As a result, the task cannot access OBS.
+Possible Causes
An underlying interface of ECS is abnormal.
+Handling Procedure
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Guardian for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-45743 Failed to Call the IAM API
+Alarm Description
This alarm is generated when Guardian fails to call the IAM API to obtain a temporary AK/SK.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
45743 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+
RoleName + |
+Specifies the role for which the alarm was generated. + |
+
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Impact on the System
The task may fail to obtain the temporary AK/SK for accessing OBS. As a result, the task cannot access OBS.
+Possible Causes
The IAM service is abnormal.
+Handling Procedure
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Guardian for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None
+ALM-50201 Doris Service Unavailable
+Alarm Description
The alarm module checks the Doris service status every 60 seconds. This alarm is generated when the alarm module detects that all FE and BE instances are abnormal.
+This alarm is cleared when any FE or BE instance recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50201 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
FusionInsight Manager cannot be used to perform cluster operations on the Doris service, and Doris service functions are unavailable.
+Possible Causes
The FE and BE instances are abnormal.
+Handling Procedure
Restart the Doris service.
+- Log in to FusionInsight Manager and choose Cluster > Services > Doris.
- On the page that is displayed, click More and select Restart Service. In the displayed dialog box, verify the password and click OK to restart the Doris service. After the service is started, go to 3.+
During the restart of the Doris service, the Doris service is unavailable and cannot provide services for external systems. Tasks connected to the Doris service fail.
+ - On FusionInsight Manager, choose O&M > Alarm > Alarms. In the alarm list, check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50202 FE CPU Usage Exceeds the Threshold
+Alarm Description
The system checks the CPU usage of the FE instance every 30 seconds. The CPU usage has a default threshold. This alarm is generated when the CPU usage exceeds the threshold (95% by default) for multiple consecutive times (3 by default).
+This alarm is cleared when Trigger Count is 1 and the CPU usage is less than or equal to the threshold. This alarm is cleared when Trigger Count is greater than 1 and the CPU usage is less than or equal to 85% of the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50202 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Processes respond slowly or do not work.
+Possible Causes
The alarm threshold or alarm trigger count is improperly configured.
+Handling Procedure
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, select the name of the desired cluster, and choose Doris > CPU and Memory > CPU Usage of FE (FE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.+
Trigger Count specifies how many times the threshold can be hit before an alarm is generated.
+ - Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50203 FE Memory Usage Exceeds the Threshold
+Alarm Description
The system checks the memory usage of the FE instance every 30 seconds. This alarm is generated when the memory usage exceeds the threshold (95% by default) for multiple consecutive times (3 by default).
+This alarm is cleared when Trigger Count is 1 and the memory usage is less than or equal to the threshold. This alarm is cleared when Trigger Count is greater than 1 and the memory usage is less than or equal to 85% of the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50203 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Processes respond slowly or do not work.
+Possible Causes
- The alarm threshold or alarm trigger count is improperly configured.
Handling Procedure
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, select the name of the desired cluster, and choose Doris > CPU and Memory > Memory Usage of FE (FE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.+
Trigger Count specifies how many times the threshold can be hit before an alarm is generated.
+ - Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50205 BE CPU Usage Exceeds the Threshold
+Alarm Description
The system checks the CPU usage of the BE instance every 30 seconds. This alarm is generated when the CPU usage exceeds the threshold (95% by default) for multiple consecutive times (3 by default).
+This alarm is cleared when Trigger Count is 1 and the CPU usage is less than or equal to the threshold. This alarm is cleared when Trigger Count is greater than 1 and the CPU usage is less than or equal to 85% of the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50205 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Processes respond slowly or do not work.
+Possible Causes
The alarm threshold or alarm trigger count is improperly configured.
+Handling Procedure
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > CPU and Memory > CPU Usage of BE (BE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.+
Trigger Count specifies how many times the threshold can be hit before an alarm is generated.
+ - Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50206 BE Memory Usage Exceeds the Threshold
+Alarm Description
The system checks the memory usage of the BE instance every 30 seconds. This alarm is generated when the memory usage exceeds the threshold for multiple consecutive times (3 by default).
+This alarm is cleared when Trigger Count is 1 and the memory usage is less than or equal to the threshold. This alarm is cleared when Trigger Count is greater than 1 and the memory usage is less than or equal to 85% of the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50206 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Processes respond slowly or do not work.
+Possible Causes
The alarm threshold or alarm trigger count is improperly configured.
+Handling Procedure
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > CPU and Memory > Memory Usage of BE (BE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.+
Trigger Count specifies how many times the threshold can be hit before an alarm is generated.
+ - Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50207 Ratio of Connections to the FE MySQL Port to the Maximum Connections Allowed Exceeds the Threshold
+Alarm Description
The system checks the number of MySQL port connections every 30 seconds. This alarm is generated when the ratio of the number of current connections to the maximum number of FE port connections exceeds the threshold (95% by default). The maximum number of FE port connections in the current cluster is specified by the qe_max_connection parameter. The default value is 1024.
+This alarm is cleared when the number of MySQL port connections on the FE node is less than or equal to the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50207 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Processes respond slowly or do not work.
+Possible Causes
- After the MySQL client is connected to Doris, the connection is not closed.
- A large number of services are concurrently connected to Doris.
Handling Procedure
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M, and click Alarm > Thresholds in the navigation pane on the left. Click the name of the desired cluster > Doris > Connection > FE MySQL Port Connections (FE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.+
If there are a large number of connections, ensure there are only necessary connections. Otherwise, the service performance may be degraded or even the service may be unavailable.
+ - Wait for 2 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50208 Failures to Clear Historical Metadata Image Files Exceed the Threshold
+Alarm Description
The system checks the number of failures to clear historical metadata image files on the FE node every 30 seconds. This alarm is generated when the number of failures exceeds the threshold (1 by default).
+This alarm is cleared when the system detects that the number of failures is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50208 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Doris metadata occupies more and more disk space, which may cause service exceptions.
+Possible Causes
The Doris service is abnormal.
+Handling Procedure
Check whether the Doris service is normal.
+- Log in to FusionInsight Manager and choose Cluster > Services > Doris.
- Check whether Running Status of the Doris service is Normal. +
- If the service process is not started, start it first and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
- Check whether other Doris-related alarms are generated in the cluster. If yes, clear them by referring to the alarm help. Then, check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
You need to manually clear the alarm after the fault is rectified.
+Related Information
None.
+ALM-50209 Failures to Generate Metadata Image Files Exceed the Threshold
+Alarm Description
The system checks the number of failures to generate metadata image files on the FE node every 30 seconds. This alarm is generated when the number of failures exceeds the threshold (1 by default).
+This alarm is cleared when the system detects that the number of failures is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50209 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The non-master FE node cannot receive the latest metadata image file. As a result, the system reliability deteriorates.
+Possible Causes
The Doris service is abnormal.
+Handling Procedure
Check the Doris service status.
+- Log in to FusionInsight Manager and choose Cluster > Services > Doris.
- Check whether Running Status of the Doris service is Normal. +
- If the service process is not started, start it first.
- Check whether other alarms are generated in the cluster. If yes, clear the alarms by referring to the alarm help. Then, check whether this alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
You need to manually clear the alarm after the fault is rectified.
+Related Information
None.
+ALM-50210 Maximum Compaction Score of All BE Nodes Exceeds the Threshold
+Alarm Description
The system checks the maximum compaction score of all BE nodes every 30 seconds. This alarm is generated when the maximum compaction score exceeds the threshold (10 by default).
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50210 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Query or write may be delayed.
+Possible Causes
The number of concurrent service requests is large in the cluster, or the compaction queue is small.
+Handling Procedure
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > Performance > Maximum compaction score of all BE nodes (BE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 2 minutes and check whether the alarm is cleared in the alarm list.
- If yes, no further action is required.
- If no, go to 5.
- Choose Cluster > Services > Doris > Configurations > All Configurations > BE(Role) > Customization, add the max_base_compaction_threads parameter to be.conf with a value of 10, and add the max_cumu_compaction_threads parameter with a value 20.
- Click Save. Click Instances, select the BE instances whose configuration has expired, click More, and select Restart Instance to restart the Doris BE instances.+
During BE instance restart, the tasks running on BE nodes will fail. The tasks on BE nodes that are not restarted are not affected.
+ - Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50211 FE Queue Length of BE Periodic Report Tasks Exceeds the Threshold
+Alarm Description
The system checks the queue length of each BE periodic report task on FE every 30 seconds. This alarm is generated when the queue length exceeds the threshold (10 by default). This value indicates the number of report tasks waiting on the master FE node. A large value indicates a poor FE processing capability.
+This alarm is cleared when the system detects that the queue length of BE periodic report tasks on FE is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50211 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The processing capability of FE is insufficient, affecting the service query speed.
+Possible Causes
The processing capability of the master FE node is insufficient due to a large number of concurrent service requests in the Doris cluster or insufficient memory for FE processes.
+Handling Procedure
Check the GC duration.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms. In the alarm list, view the role name and obtain the IP address of the instance in Location of the alarm whose ID is 50211.
- Choose Cluster > Services > Doris > Instances, click the FE instance for which the alarm is generated, and click the Chart tab of the instance. +
- Choose Cluster > Services > Doris > Configurations > All Configurations > FE(Role) > JVM, and increase the value of -Xmx in FE_GC_OPTS. The default value is 8GB.
- If this alarm is generated occasionally, increase the value by 0.5 times. If this alarm is generated frequently, double the parameter value.
- In the case of large service volume and high service concurrency, you are advised to add instances.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > Queue > Queue Length of BE Periodic Report Tasks on the FE (FE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50212 Accumulated Old-Generation GC Duration of the FE Process Exceeds the Threshold
+Alarm Description
The system checks the accumulated old-generation GC duration of the FE process every 30 seconds. This alarm is generated when the accumulated GC duration exceeds the threshold (3000 ms by default).
+This alarm is cleared when the system detects that the accumulated old-generation GC duration of the FE process is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50212 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
A long GC duration of the FE process may interrupt the services.
+Possible Causes
The heap memory of the FE process is overused or inappropriately allocated, causing frequent occurrence of the GC process.
+Handling Procedure
Check the GC duration.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms. In the alarm list, view the role name and obtain the IP address of the instance in Location of the alarm whose ID is 50212.
- Choose Cluster > Services > Doris > Instances, click the FE instance for which the alarm is generated, and click the Chart tab of the instance. +
- Choose Cluster > Services > Doris > Configurations > All Configurations > FE(Role) > JVM, and increase the value of -Xmx in FE_GC_OPTS. The default value is 8G.
- If this alarm is occasionally generated, increase the value by 0.5 times. If this alarm is frequently generated, double the value.
- In the case of large service volume and high service concurrency, you are advised to add instances.
- Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50213 Number of Tasks Queuing in the FE Thread Pool for Interacting with BE Exceeds the Threshold
+Alarm Description
The system checks the number of queuing tasks in the FE thread pool for interacting with BE every 30 seconds. This alarm is generated when the number of queuing tasks exceeds the threshold (10 by default). This FE thread pool is the working thread pool of ThriftServer. It is specified by rpc_port in the fe.conf file and is used to interact with BE.
+This alarm is cleared when the system detects that the number of tasks queuing in the FE thread pool for interacting with BE is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50213 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The read and write of the Doris service slows down.
+Possible Causes
There are a large number of concurrent service requests, causing too many queuing tasks.
+Handling Procedure
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > Queue > Number of tasks that are queuing in the thread pool for interaction between the FE and the BE (FE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50214 Number of Tasks Queuing in the FE Thread Pool for Task Processing Exceeds the Threshold
+Alarm Description
The system checks the number of queuing tasks in the FE thread pool for processing tasks every 30 seconds. This alarm is generated when the number of queuing tasks exceeds the threshold (10 by default). This thread pool is used by the NIO MySQL Server to process tasks.
+This alarm is cleared when the system detects that the number of tasks queuing in the FE thread pool for processing tasks is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50214 + |
+Minor + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The task execution of the entire system becomes slow and blocked.
+Possible Causes
Large tasks may block the task execution of the queue.
+Handling Procedure
Check the execution status of FE tasks.
+- On FusionInsight Manager, choose Cluster > Services > Doris. Click the Chart tab, select Connection from Chart Category in the left pane, and view the FE MySQL Port Connections chart. If the number of connections is large, click Instances, select the FE instance, and click the Chart tab. Select CPU and Memory from Chart Category and view the CPU Usage of FE chart. If the CPU usage is high, check the Time field in FE audit log /var/log/Bigdata/audit/doris/fe/fe.audit.log to collect statistics on the average task duration. If the value is also high, the alarm is caused by large concurrent tasks.
- After connecting to Doris, run the following command to check whether the default value of queryTimeout is too large. The default value is 300 seconds.
show variables like 'query_timeout';
+ + - Run the following command to shorten the timeout period based on site requirements to block the tasks that take a long time:
set global query_timeout=xxx;
+ - Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > Queue > Queue Length of Query Execution Thread Pool (BE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 10 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50215 Longest Duration of RPC Requests Received by Each FE Thrift Method Exceeds the Threshold
+Alarm Description
The system checks the longest duration of RPC requests received by each FE Thrift method every 30 seconds. This alarm is generated when the longest duration exceeds the threshold (5000 ms by default).
+This alarm is cleared when the longest duration of RPC requests received by each FE Thrift method is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50215 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
A longer RPC duration indicates a higher performance load and slower network request processing, which may cause service congestion.
+Possible Causes
- The network has a latency.
- There are too many concurrent large SQL tasks.
Handling Procedure
- Log in to the host where the faulty node is deployed as user root and run ping IP addresses of all Doris nodes to check whether the peer host can be pinged. +
- Contact the network administrator to restore the network.
- On FusionInsight Manager, choose Cluster > Services > Doris. Click the Chart tab, select Connection from Chart Category in the left pane, and view the FE MySQL Port Connections chart. If the number of connections is large, click Instances, select the FE instance, and click the Chart tab. Select CPU and Memory from Chart Category and view the CPU Usage of FE chart. If the CPU usage is high, check the Time field in FE audit log /var/log/Bigdata/audit/doris/fe/fe.audit.log to collect statistics on the average task duration. If the value is also high, the alarm is caused by large concurrent tasks.
- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > Performance > Longest duration of RPC requests received by each method of the FE thrift interface (FE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 10 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 8.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50216 Memory Usage of the FE Node Exceeds the Threshold
+Alarm Description
The system checks the memory usage of the FE node every 30 seconds. This alarm is generated when the memory usage exceeds the threshold (95% by default).
+This alarm is cleared when the memory usage of the FE node falls below the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50216 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Task execution and client connection to the FE are affected.
+Possible Causes
The FE heap memory is too small.
+Handling Procedure
Check the FE heap memory usage.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, select the name of the desired cluster, and choose Doris > CPU and Memory > Memory usage of the FE node (FE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Log in to the FE node for which the alarm is generated as user omm, run the top command to check the memory usage of processes, locate the process with high memory usage, and check whether the process belongs to the current service and is running properly.
- If yes, go to 3.
- If no, isolate or stop the process, or adjust the memory size, and check whether the memory is released.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
+ - During the restart of the Doris service, the Doris service is unavailable and cannot provide services for external systems. Tasks connected to the Doris service fail.
- During instance restart, the tasks running on the nodes of the instance will fail. The tasks on instance nodes that are not restarted are not affected.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50217 Heap Memory Usage of the FE Node Exceeds the Threshold
+Alarm Description
The system checks the heap memory usage of the FE node every 30 seconds. This alarm is generated when the heap memory usage exceeds the threshold (95% by default).
+This alarm is cleared when the heap memory usage of the FE node falls below the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50217 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Task execution and client connection to the FE are affected.
+Possible Causes
The FE heap memory is too small.
+Handling Procedure
Check heap memory usage.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, select the name of the desired cluster, and choose Doris > CPU and Memory > Heap memory usage of the FE node (FE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- On FusionInsight Manager, choose Cluster > Services > Doris > FE > Configurations > All Configurations, search for the FE_GC_OPTS parameter, increase the value of -Xmx as required, click Save, and click OK.+
- If this alarm is generated, the heap memory configured for the current Doris instance is not enough for data transmission. You are advised to open the instance monitoring page, display the Doris heap memory resource status monitoring chart, and observe the change trend of the heap memory used by Doris in the monitoring chart. Then change the value of -Xmx to twice the current heap memory usage or to another value to meet site requirements.
- When setting the heap memory, you can set -Xms and -Xmx to approximately the same value to prevent performance deterioration caused by heap size adjustment after each GC.
- The sum of -Xmx and XX:MaxPermSize cannot be greater than the actual physical memory of the node server.
- Restart the affected services or instances and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
+ - During the restart of the Doris service, the Doris service is unavailable and cannot provide services for external systems. Tasks connected to the Doris service fail.
- During instance restart, the tasks running on the nodes of the instance will fail. The tasks on instance nodes that are not restarted are not affected.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50219 Length of the Queue in the Thread Pool for Query Execution Exceeds the Threshold
+Alarm Description
The system checks the length of the waiting queue in the query execution thread pool every 30 seconds. This alarm is generated when the length exceeds the threshold (20 by default).
+This alarm is cleared when the length of the waiting queue in the current query execution thread pool is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50219 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The task execution of the entire system becomes slow and blocked.
+Possible Causes
Large tasks may block the task execution of the queue.
+Handling Procedure
Check the execution status of tasks.
+- On FusionInsight Manager, choose Cluster > Services > Doris. Click the Chart tab, select Connection from Chart Category in the left pane, and view the FE MySQL Port Connections chart. If the number of connections is large, click Instances, select the FE instance, and click the Chart tab. Select CPU and Memory from Chart Category and view the CPU Usage of FE chart. If the CPU usage is high, check the Time field in FE audit log /var/log/Bigdata/audit/doris/fe/fe.audit.log to collect statistics on the average task duration. If the value is also high, the alarm is caused by large concurrent tasks.
- After connecting to Doris, run the following command to check the queryTimeout value of the system:
show variables like 'query_timeout';
+If the value is too large, run the set global query_timeout=xxx; command to shorten the timeout interval and block tasks that last for a long time.
+ - Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > Queue > Queue Length of Query Execution Thread Pool (BE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 10 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 7.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50220 Error Rate of TCP Packet Receiving Exceeds the Threshold
+Alarm Description
The system checks the rate of TCP packet receiving errors every 30 seconds. This alarm is generated when the error rate exceeds the threshold (5% by default).
+This alarm is cleared when the error rate is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50220 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The task fails or data is lost.
+Possible Causes
The network is faulty, so data cannot be sent.
+Handling Procedure
- Log in to the host where the faulty node is deployed as user root and run ping IP addresses of all Doris nodes to check whether the peer host can be pinged. +
- Contact the network administrator to restore the network.
- Wait for a while and check whether the alarm is cleared in the alarm list.
- If yes, no further action is required.
- If no, go to 4.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50221 BE Data Disk Usage Exceeds the Threshold
+Alarm Description
The system checks the usage of BE data disks every 30 seconds. This alarm is generated when the disk usage exceeds the threshold (95% by default).
+This alarm is cleared when the system detects that the disk usage is less than the threshold.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50221 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
New data fails to be written, and the task is interrupted.
+Possible Causes
- The disk space of the cluster is full.
- Data skew occurs among BE nodes.
Handling Procedure
- Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
- Expand the disk capacity of the node for which the alarm is generated.
- Go to 4 if the expansion fails or the alarm persists after the expansion.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50222 Disk Status of a Specified Data Directory on BE Is Abnormal
+Alarm Description
The system checks the disk status of a specified data directory on BE every 30 seconds. This alarm is generated when the disk status is not 1 (1 indicates the normal state and 0 indicates the abnormal state). This alarm is cleared when the disk status of the specified data directory on BE becomes normal.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50222 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
Service data may be unavailable.
+Possible Causes
- The hard disk is faulty.
- The disk permissions are set incorrectly.
Handling Procedure
Check whether a disk alarm is generated.
+- On FusionInsight Manager, choose O&M > Alarm > Alarms and check whether ALM-12014 Partition Lost or ALM-12033 Slow Disk Fault exists. +
- Rectify the fault by referring to the handling procedure of ALM-12014 Partition Lost or ALM-12033 Slow Disk Fault. Then, check whether the alarm is cleared. +
- Wait 5 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
Modify disk permissions.
+- Choose O&M > Alarm > Alarms and view Location and Additional Information of the alarm to obtain the location of the faulty disk.
- Log in to the node for which the alarm is generated as user root. Go to the directory where the faulty disk is located, and run the ll command to check whether the permission for the faulty disk is 711 and whether the user is omm. +
- Modify the permission of the faulty disk. For example, if the faulty disk is data1, run the following commands:
chown omm:wheel data1
+chmod 711 data1
+ - Choose Cluster > Services > Doris > Instances, select this BE instance, click More, and select Restart Instance. Wait 5 minutes and check whether an alarm is generated.
- If no, no further action is required.
- If yes, go to 8.
+ During BE instance restart, the tasks running on BE nodes will fail. The tasks on BE nodes that are not restarted are not affected.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris and OMS for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50223 Maximum Memory Required by BE Is Greater Than the Remaining Memory of the Machine
+Alarm Description
The system checks whether the maximum memory required by BE is greater than the available memory every 30 seconds. This alarm is generated when the value is not 1 (1 indicates that the maximum required memory is less than or equal to the available memory, and 0 indicates that the maximum required memory is greater than the available memory).
+This alarm is cleared when the maximum required memory is less than or equal to the available memory.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50223 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
A task may fail to apply for memory when running.
+Possible Causes
Too much BE node memory has been occupied by other processes, or the maximum memory set for the BE service is too large.
+Handling Procedure
Check whether the maximum memory set for the BE node is proper.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > CPU and Memory > Relationship between the maximum memory size of the BE and the remaining memory size of the machine (BE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 5.
- Log in to the BE node for which the alarm is generated as user omm, run the top command to check the memory usage of processes, locate the process with high memory usage, and check whether the process belongs to the current service and is running properly.
- If yes, go to 6.
- If no, isolate or stop the process, or adjust the memory size, and check whether the memory is released.
- Log in to the BE node for which the alarm is generated as user omm and run the free -g command to check the total memory and remaining memory in the system and estimate the memory usage.
- On FusionInsight Manager, choose Cluster > Services > Doris > Configurations > All Configurations > BE(Role) > Memory and decrease the value of mem_limit. This parameter specifies the maximum memory allowed for BE. Then save the modification and restart the BE instance.+
During BE instance restart, the tasks running on BE nodes will fail. The tasks on BE nodes that are not restarted are not affected.
+ - After the BE instance is restarted, wait 5 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 9.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select Doris for the target cluster.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50224 Failures a Certain Task Type on BE Are Increasing
+Alarm Description
The system checks whether the number of failed tasks of a certain type on BE is increasing every 30 seconds. This alarm is generated when the system detects that the value is not 1 (1 indicates that the number of failed tasks of a certain type does not increase, and 0 indicates that the failed tasks of a certain type are increasing).
+This alarm is cleared when the system detects that the number of failed tasks of a certain type on BE does not increase.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50224 + |
+Major + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
A task fails to be executed repeatedly in a certain scenario.
+Possible Causes
A BE exception may occur. As a result, the number of failed tasks increases in a certain scenario.
+Handling Procedure
Check whether the alarm threshold or alarm trigger count is properly configured.
+- Log in to FusionInsight Manager, choose O&M > Alarm > Thresholds, click the name of the desired cluster, and choose Doris > Exception > Check whether the number of failed tasks of a certain type increases (BE).
- Click the edit button next to Trigger Count, change the number based on site requirements, and click OK.
- Click Modify in the Operation column, change the alarm threshold based on site requirements, and click OK.
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 5.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
You need to manually clear the alarm after the fault is rectified.
+Related Information
None.
+ALM-50225 FE Instance Fault
+Alarm Description
The system checks the FE process status every 30 seconds. This alarm is generated when the value is greater than 0 (0 indicates that the FE process is normal and 1 indicates that the FE process is abnormal).
+This alarm is cleared when the system detects that the FE process becomes normal.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50225 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The current FE instance is unavailable.
+Possible Causes
The FE instance is faulty or restarted.
+Handling Procedure
- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. In the alarm list, view the role name and obtain the IP address of the instance in Location of the alarm whose ID is 50225.
- Choose Cluster > Services > Doris > Instances, click the FE instance for which the alarm is generated, and check whether Running Status of the instance is Restoring. +
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 4.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50226 BE Instance Fault
+Alarm Description
The system checks the BE process status every 30 seconds. This alarm is generated when the value is greater than 0 (0 indicates that the BE process is normal and 1 indicates that the BE process is abnormal).
+This alarm is cleared when the system detects that the BE process becomes normal.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50226 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Trigger Condition + |
+Specifies the threshold for triggering the alarm. + |
+
Impact on the System
The current BE instance is unavailable.
+Possible Causes
The BE instance is faulty or restarted.
+Handling Procedure
- Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. In the alarm list, view the role name and obtain the IP address of the instance in Location of the alarm whose ID is 50226.
- Choose Cluster > Services > Doris > Instances, click the BE instance for which the alarm is generated, and check whether Running Status of the instance is Restoring. +
- Wait 2 minutes and check whether the alarm is automatically cleared.
- If yes, no further action is required.
- If no, go to 4.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, select Doris for the target cluster, and click OK.
- Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50401 Number of JobServer Jobs Waiting to Be Executed Exceeds the Threshold
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks the number of jobs submitted to JobServer every 30 seconds. This alarm is generated when the number of jobs to be executed exceeds 800.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50401 + |
+Critical (default threshold: 900) +Major (default threshold: 800) + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm was generated. + |
+
ServiceName + |
+Specifies the service for which the alarm was generated. + |
+
RoleName + |
+Specifies the role for which the alarm was generated. + |
+
HostName + |
+Specifies the host for which the alarm was generated. + |
+
Impact on the System
Too many JobServer jobs are detected in the queue. For details about the queue usage, see the Additional Information field of this alarm. The impacts are as follows:
+- When the number of JobServer jobs in the queue reaches the maximum (1000 by default), new jobs cannot be added.
- Before the number of JobServer jobs in the queue reaches the maximum, new JobServer jobs cannot be submitted quickly. For example, it takes more time (even hours) to submit added jobs or add new jobs to Yarn.
Possible Causes
Too many jobs are submitted instantaneously.
+Handling Procedure
- Log in to FusionInsight Manager and choose Cluster > Services > JobGateway.
- Click the Instances tab, click Add Instance, and add JobServer instances based on the number of submitted jobs.
- After the instances are added, restart the JobGateway service.+
The job functions of JobGateway will become unavailable during the service restart.
+ - Wait 5 minutes and check whether the alarm is automatically cleared.
If yes, no further action is required.
+If no, go to 5.
+
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select JobGateway for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
- Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+ALM-50402 JobGateway Service Unavailable
+ This section applies only to MRS 3.3.0 or later.
+Alarm Description
The system checks the JobGateway service status every 60 seconds. This alarm is generated when the JobGateway service is abnormal.
+This alarm is cleared when the JobGateway service recovers.
+Alarm Attributes
+Alarm ID + |
+Alarm Severity + |
+Auto Cleared + |
+
|---|---|---|
50402 + |
+Critical + |
+Yes + |
+
Alarm Parameters
+Parameter + |
+Description + |
+
|---|---|
Source + |
+Specifies the cluster or system for which the alarm is generated. + |
+
ServiceName + |
+Specifies the service for which the alarm is generated. + |
+
RoleName + |
+Specifies the role for which the alarm is generated. + |
+
HostName + |
+Specifies the host for which the alarm is generated. + |
+
Impact on the System
No job submission operation can be performed on JobGateway in the cluster. The components that depend on JobGateway in the cluster will become faulty.
+Possible Causes
The node where the JobGateway service locates is faulty.
+Handling Procedure
- Log in to FusionInsight Manager, choose Cluster > Services > JobGateway, and click the Instance tab. Check for JobServer or JobBalancer instances that are faulty or not started and view the host names of these instances.
- On the Alarm page of FusionInsight Manager, check whether the NodeAgent Process Is Abnormal alarm is generated. +
- Check whether the host name in the alarm information is the same as the host name in 1. +
- Clear the alarm by following the instructions provided in ALM-12006 NodeAgent Process Is Abnormal.
- In the alarm list, check whether alarm JobGateway Service Unavailable is cleared.
- If yes, no further action is required.
- If no, go to 6.
Collect fault information.
+- On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
- Expand the Service drop-down list, and select JobGateway for the target cluster.
- Click
in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download. - Contact O&M personnel and provide the collected logs.
Alarm Clearance
This alarm is automatically cleared after the fault is rectified.
+Related Information
None.
+You can view the data source of a monitoring chart in the lower left corner of the chart. You can zoom in on a monitoring report to view chart values more clearly or close the monitoring report.
Alarm Analysis
On the Alarm Analysis tab page, you can view the Top 20 Alarms table and Analysis on Top 3 Alarms chart. You can click an alarm name in the Top 20 Alarms table to view analysis information of this alarm only. Alarm analysis allows you to view top alarms and their occurrence time so you can handle alarms accordingly, improving system stability.
+ For MRS 3.3.0 and later versions, the alarm information page is different from that of other versions.
+Scenario
On MRS Manager, you can customize monitoring items to display on the homepage and export monitoring data.
The interval on the horizontal axis of the chart varies depending on the time period you specify. Data monitoring rules are as follows:
-- 0 to 25 hours: The interval is 5 minutes. The cluster must have been installed for at least 10 minutes, and monitoring data of a maximum of 15 days is saved.
- 25 to 150 hours: The interval is 30 minutes. The cluster must have been installed for at least 30 minutes, and monitoring data of a maximum of 3 months is saved.
- 150 to 300 hours: The interval is 1 hour. The cluster must have been installed for at least 1 hour, and monitoring data of a maximum of 3 months is saved.
- 300 hours to 300 days: The interval is 1 day. The cluster must have been installed for at least 1 day, and monitoring data of a maximum of 6 months is saved.
- Over 300 days: The interval is 7 days. The cluster must have been installed for more than 7 days, and monitoring data of a maximum of 1 year is saved.
- If the disk usage of the partition where GaussDB resides exceeds 80%, real-time monitoring data and monitoring data whose interval is 5 minutes will be deleted.
- Storage resources (HDFS) in Tenant Resources (0 to 300 hours): The interval is 1 hour. The cluster must have been installed for at least 1 hour, and monitoring data of a maximum of 3 months is saved.
- If the disk usage of the partition where GaussDB resides exceeds 80%, real-time monitoring data and monitoring data whose interval is 5 minutes will be deleted.
- Storage resources (HDFS) in Tenant Resources (0 to 300 hours): The interval is 1 hour. The cluster must have been installed for at least 1 hour, and monitoring data of a maximum of 3 months is saved.
For clusters of versions earlier than MRS 3.3.0:
+- 0 to 25 hours: The interval is 5 minutes. The cluster must have been installed for at least 10 minutes, and monitoring data of a maximum of 15 days is saved.
- 25 to 150 hours: The interval is 30 minutes. The cluster must have been installed for at least 30 minutes, and monitoring data of a maximum of 3 months is saved.
- 150 to 300 hours: The interval is 1 hour. The cluster must have been installed for at least 1 hour, and monitoring data of a maximum of 3 months is saved.
- 300 hours to 300 days: The interval is 1 day. The cluster must have been installed for at least 1 day, and monitoring data of a maximum of 6 months is saved.
- Over 300 days: The interval is 7 days. The cluster must have been installed for more than 7 days, and monitoring data of a maximum of 1 year is saved.
For clusters of MRS 3.3.0 or later versions:
+- 0 to 21 hours and 20 minutes: The interval is 5 minutes. The cluster must have been installed for at least 10 minutes, and monitoring data of a maximum of 90 days is saved.
- 21 hours and 20 minutes to 128 hours: The interval is 30 minutes. The cluster must have been installed for at least 30 minutes, and monitoring data of a maximum of 90 days is saved.
- 128 to 256 hours: The interval is 1 hour. The cluster must have been installed for at least 1 hour, and monitoring data of a maximum of 90 days is saved.
- 256 hours to 256 days: The interval is 1 day. The cluster must have been installed for at least 1 day, and monitoring data of a maximum of 90 days is saved.
Customizing a Monitoring Metric Report
- Log in to MRS Manager.
- Choose Homepage.
- In the upper right corner of the chart area, click
and choose Customize from the displayed menu.diff --git a/docs/mrs/umn/admin_guide_000011.html b/docs/mrs/umn/admin_guide_000011.html index 228fdb59..1c556623 100644 --- a/docs/mrs/umn/admin_guide_000011.html +++ b/docs/mrs/umn/admin_guide_000011.html @@ -4,72 +4,74 @@ Monitoring data of the past 1 hour is displayed at an interval of 5 minutes. After you enter the Real-time Monitoring page, you can view that real-time monitoring data is displayed on the right of the monitoring chart at an interval of 5 minutes.
Dashboard
Log in to MRS Manager and choose Cluster > Name of the desired cluster > Dashboard to view the status of the current cluster.
On the Dashboard tab page, you can start, stop, perform a rolling restart of, synchronize configurations to, and perform other basic operations on the current cluster.
+- For MRS 3.3.0 or later, the Cluster > Dashboard page has been removed from Manager. You can choose More in the upper right corner of the Homepage to access cluster maintenance and management functions.
+Table 1 Maintenance and management operations UI Portal
+Table 1 Maintenance and management operations UI Portal
Description
+Description
+
Starts all services in the cluster.
+Starts all services in the cluster.
+
Stops all services in the cluster.
+Stops all services in the cluster.
More > Restart
+More > Restart
Restarts all services in the cluster.
+Restarts all services in the cluster.
More > Rolling-restart Service
+More > Rolling-restart Service
Restarts all services in the cluster one at a time without interrupting workloads. For details about how to perform a rolling restart, see Performing a Rolling Restart of a Cluster.
+Restarts all services in the cluster one at a time without interrupting workloads. For details about how to perform a rolling restart, see Performing a Rolling Restart of a Cluster.
More > Synchronize Configurations
+More > Synchronize Configurations
Enables new configuration parameters for all services in the cluster.
+Enables new configuration parameters for all services in the cluster.
More > Restart Configuration-Expired Instances
+More > Restart Configuration-Expired Instances
Restarts expired instances for all services in the cluster. For details, see Managing Expired Configurations.
+Restarts expired instances for all services in the cluster. For details, see Managing Expired Configurations.
More > Health Check
+More > Health Check
Performs a health check on the OMS nodes, all services, and the rest nodes in the cluster. There are three types of check items: running status, related alarms, and custom monitoring metrics. The health check results are not always the same as the values of Running Status displayed on the GUI.
+Performs a health check on the OMS nodes, all services, and the rest nodes in the cluster. There are three types of check items: running status, related alarms, and custom monitoring metrics. The health check results are not always the same as the values of Running Status displayed on the GUI.
You can export check results by clicking Export in the upper left corner of the checklist. If any issues are detected, you can click View Help to find a troubleshooting method.
More > Download Client
+More > Download Client
Downloads the default client. For details, see Downloading the Client.
+Downloads the default client. For details, see Downloading the Client.
More > Export Installation Template
+More > Export Installation Template
Batch exports all installation configurations of the cluster, such as the cluster authentication mode, node information, and service configuration. You can use this function when you need to reinstall the cluster in the same environment.
+Batch exports all installation configurations of the cluster, such as the cluster authentication mode, node information, and service configuration. You can use this function when you need to reinstall the cluster in the same environment.
More > Export Configurations
+More > Export Configurations
Batch exports configurations of all services in the cluster.
+Batch exports configurations of all services in the cluster.
More > Enter Maintenance Mode and More > Exit Maintenance Mode
+More > Enter Maintenance Mode and More > Exit Maintenance Mode
Enters or exits the cluster maintenance mode.
+Enters or exits the cluster maintenance mode.
More > O&M View
+More > O&M View
Allows you to view services or hosts that are in the maintenance mode.
+Allows you to view services or hosts that are in the maintenance mode.
-Impact on the System
A rolling restart takes a longer time and may affect service throughput and performance.
Procedure
- Log in to MRS Manager.
- Choose Cluster > Name of the target cluster > Dashboard. On this tab page, choose More > Rolling-restart Service.
- In the dialog box that is displayed, enter the password of the current login user and click OK.
- Configure the parameters based on site requirements.
+
Procedure
- Log in to MRS Manager.
- Choose Cluster > Name of the target cluster > Dashboard. On this tab page, choose More > Rolling-restart Service.+
For MRS 3.3.0 or later, the Cluster > Dashboard page has been removed from Manager. You can choose More in the upper right corner of the Homepage to access cluster maintenance and management functions.
+ - In the dialog box that is displayed, enter the password of the current login user and click OK.
- Configure the parameters based on site requirements.
Table 1 Rolling restart parameters Parameter
Description
diff --git a/docs/mrs/umn/admin_guide_000013.html b/docs/mrs/umn/admin_guide_000013.html index 29b78a56..98f8ee7e 100644 --- a/docs/mrs/umn/admin_guide_000013.html +++ b/docs/mrs/umn/admin_guide_000013.html @@ -7,7 +7,9 @@Impact on the System
- After synchronizing the cluster configuration, you need to restart the services whose configuration has expired. These services are unavailable during restart.
- The instances whose configuration has expired are unavailable during restart.
Procedure
Synchronize the configuration.
-- Log in to MRS Manager.
- Choose Cluster > Name of the desired cluster > Dashboard.
- On this page, choose More > Synchronize Configuration.
- In the dialog box that is displayed, click OK.
- Log in to MRS Manager.
- Choose Cluster > Name of the desired cluster > Dashboard.+
For MRS 3.3.0 or later, the Cluster > Dashboard page has been removed from Manager. You can choose More in the upper right corner of the Homepage to access cluster maintenance and management functions.
+ - On this page, choose More > Synchronize Configuration.
- In the dialog box that is displayed, click OK.
Restart configuration-expired instances.
- Choose More > Restart Configuration-Expired Instances.
- In the dialog box that is displayed, enter the password of the current login user and click OK.
- In the displayed dialog box, click OK.
You can click View Instance to open the list of all expired instances and confirm that the instances have been restarted.
Downloading the Client
-Scenario
Use the default client provided by MRS clusters to manage the cluster, run services, and perform secondary development. Before you use this client, you need to download its software package.
Procedure
- Log in to MRS Manager.
- Choose Cluster > Name of the desired cluster > Dashboard. On the page that is displayed, choose More > Download Client.
The Download Cluster Client dialog box is displayed.
+Procedure
- Log in to MRS Manager.
- Choose Cluster > Name of the desired cluster > Dashboard. On the page that is displayed, choose More > Download Client.+
For MRS 3.3.0 or later, the Cluster > Dashboard page has been removed from Manager.
+The Download Cluster Client dialog box is displayed.
- Select a client type for Select Client Type.
- Complete Client: the package contains scripts, compilation files, and configuration files.
- Configuration Files Only: the package contains only the client configuration files.
This type is applicable to application development tasks. For example, after a complete client is downloaded and installed, the cluster administrator modifies the service configuration on MRS Manager, and developers need to update the client configuration files.
Set Select Platform Type to x86_64 or aarch64. To run the client on x86 nodes, select x86_64; to tun the client on TaiShan nodes, select aarch64. By default, you should select a client that has the same architecture as your servers.
diff --git a/docs/mrs/umn/admin_guide_000023.html b/docs/mrs/umn/admin_guide_000023.html index 64c783e5..a3ae1fac 100644 --- a/docs/mrs/umn/admin_guide_000023.html +++ b/docs/mrs/umn/admin_guide_000023.html @@ -4,7 +4,7 @@Scenario
The client package downloaded from MRS Manager contains the client batch upgrade tool. When multiple clients need to be upgraded after the cluster upgrade or scale-out, you can use this tool to upgrade the clients in batches with a few clicks. In addition, the tool provides the lightweight function of batch updating the /etc/hosts file on the nodes where the clients are located.
Procedure
Prepare for the client upgrade.
-- Log in to MRS Manager.
- Choose Cluster, click the name of the desired cluster, click More, and select Download Client to download the complete client to the specified directory on the server.
For details, see Downloading the Client.
+- Log in to MRS Manager.
- Choose Cluster, click the name of the desired cluster, click More, and select Download Client (for MRS 3.3.0 or later, click Download Client on the Homepage page) to download the complete client to the specified directory on the server.
For details, see Downloading the Client.
Decompress the downloaded client package and find the batch_upgrade directory, for example, /tmp/FusionInsight-Client/FusionInsight_Cluster_1_Services_ClientConfig/batch_upgrade.
- Choose Cluster, click the name of the desired cluster, and choose Client Management. On the Client Management page, click Export All to export all client information to the local PC.
- Decompress the exported client information and upload the client-info.cfg file to the batch_upgrade directory.
- Supplement the password in the client-info.cfg file by referring to Reference Information.
Upgrade clients in batches.
diff --git a/docs/mrs/umn/admin_guide_000040.html b/docs/mrs/umn/admin_guide_000040.html index 66cb1245..b27f3b8e 100644 --- a/docs/mrs/umn/admin_guide_000040.html +++ b/docs/mrs/umn/admin_guide_000040.html @@ -3,7 +3,9 @@Decommissioning and Recommissioning an Instance
Scenario
Some role instances provide services for external services in distributed and parallel mode. Services independently store information about whether each instance can be used. Therefore, you need to use MRS Manager to recommission or decommission these instances to change the instance running status.
Some instances do not support the recommissioning and decommissioning functions.
- The following roles support decommissioning and recommissioning: HDFS DataNode, YARN NodeManager, and HBase RegionServer.diff --git a/docs/mrs/umn/admin_guide_000070.html b/docs/mrs/umn/admin_guide_000070.html index ae9e5348..9887bcfa 100644 --- a/docs/mrs/umn/admin_guide_000070.html +++ b/docs/mrs/umn/admin_guide_000070.html @@ -68,6 +68,7 @@- If the number of the DataNodes is less than or equal to that of HDFS replicas, decommissioning cannot be performed. If the number of HDFS replicas is three and the number of DataNodes is less than four in the system, decommissioning cannot be performed. In this case, an error will be reported and force MRS Manager to exit the decommissioning 30 minutes after MRS Manager attempts to perform the decommissioning.
- During MapReduce task execution, files with 10 replicas are generated. Therefore, if the number of DataNode instances is less than 10, decommissioning cannot be performed.
- If the number of DataNode racks (the number of racks is determined by the number of racks configured for each DataNode) is greater than 1 before the decommissioning, and after some DataNodes are decommissioned, that of the remaining DataNodes changes to 1, the decommissioning will fail. Therefore, before decommissioning DataNode instances, you need to evaluate the impact of decommissioning on the number of racks to adjust the DataNodes to be decommissioned.
- If multiple DataNodes are decommissioned at the same time, and each of them stores a large volume of data, the DataNodes may fail to be decommissioned due to timeout. To avoid this problem, it is recommended that one DataNode be decommissioned each time and multiple decommissioning operations be performed.
The following roles support decommissioning and recommissioning: HDFS DataNode, YARN NodeManager, and HBase RegionServer.- By default, if the number of the DataNodes is less than or equal to that of HDFS replicas, decommissioning cannot be performed. If the number of HDFS replicas is three and the number of DataNodes is less than four in the system, decommissioning cannot be performed. In this case, an error will be reported and force MRS Manager to exit the decommissioning 30 minutes after MRS Manager attempts to perform the decommissioning.
- You can enable quick decommissioning before decommissioning DataNodes for clusters of MRS 3.3.0 or later. In this case, when the number of DataNodes meets the value of dfs.namenode.decommission.force.replication.min, the system decommissions the nodes and adds HDFS copies at the same time. If data is written during quick decommissioning, data may be lost. Exercise caution when performing this operation. The parameters related to quick decommissioning are listed as follows. You can search for and view the parameters on the HDFS configuration page on FusionInsight Manager.
dfs.namenode.decommission.force.enabled: whether to enable quick decommissioning for DataNode. If this parameter is set to true, the function is enabled.
+dfs.namenode.decommission.force.replication.min: minimum number of available copies of a block required for DataNode quick decommissioning. The value ranges from 1 to 3.
+ - During MapReduce task execution, files with 10 replicas are generated. Therefore, if the number of DataNode instances is less than 10, decommissioning cannot be performed.
- If the number of DataNode racks (the number of racks is determined by the number of racks configured for each DataNode) is greater than 1 before the decommissioning, and after some DataNodes are decommissioned, that of the remaining DataNodes changes to 1, the decommissioning will fail. Therefore, before decommissioning DataNode instances, you need to evaluate the impact of decommissioning on the number of racks to adjust the DataNodes to be decommissioned.
- If multiple DataNodes are decommissioned at the same time, and each of them stores a large volume of data, the DataNodes may fail to be decommissioned due to timeout. To avoid this problem, it is recommended that one DataNode be decommissioned each time and multiple decommissioning operations be performed.
Additional Information
Indicates the error information.
+MRS 3.3.0 and later versions: You can view the monitoring metric values in Additional Information if thresholds are set for the metrics to generate alarms.
Location
diff --git a/docs/mrs/umn/admin_guide_000071.html b/docs/mrs/umn/admin_guide_000071.html index 191771b1..8b0cb15d 100644 --- a/docs/mrs/umn/admin_guide_000071.html +++ b/docs/mrs/umn/admin_guide_000071.html @@ -4,7 +4,7 @@Scenario
You can configure monitoring indicator thresholds to monitor the health status of indicators on MRS Manager. If abnormal data occurs and the preset conditions are met, the system triggers an alarm and displays the alarm information on the alarm page.
Procedure
- Log in to MRS Manager.
- Choose O&M > Alarm > Thresholds.
- Select a monitoring metric for a host or service in the cluster.Figure 1 Configuring the threshold for a metric-
For example, after selecting Host Memory Usage, the information about this indicator threshold is displayed.- If the alarm sending switch is displayed as
, an alarm is triggered if the threshold is reached. - Alarm ID and Alarm Name: alarm information triggered against the threshold
- Trigger Count: MRS Manager checks whether the value of a monitoring metric reaches the threshold. If the number of consecutive checks reaches the value of Trigger Count, an alarm is generated. Trigger Count is configurable.
- Check Period (s): interval for the system to check the monitoring metric.
- The rules in the rule list are used to trigger alarms.
For example, after selecting Host Memory Usage, the information about this indicator threshold is displayed.- If the alarm sending switch is displayed as , an alarm is triggered if the threshold is reached.
- When Alarm Severity is on, hierarchical alarms are enabled. The system dynamically reports alarms at each severity based on the real-time metric values and hierarchical thresholds set for the severity. MRS 3.3.0 or later supports this function.
- Alarm ID and Alarm Name: alarm information triggered against the threshold
- Trigger Count: MRS Manager checks whether the value of a monitoring metric reaches the threshold. If the number of consecutive checks reaches the value of Trigger Count, an alarm is generated. Trigger Count is configurable.
- Check Period (s): interval for the system to check the monitoring metric.
- The rules in the rule list are used to trigger alarms.
- Click Create Rule to add rules used for monitoring indicators.
Table 1 Monitoring indicator rule parameters Parameter
@@ -72,1114 +72,1116 @@Monitoring Metric Reference
MRS Manager alarm monitoring metrics are classified as node information metrics and cluster service metrics. Table 2 describes the metrics for which you can configure thresholds on nodes.
+- On FusionInsight Manager of MRS 3.3.0 or later, alarms of some components can be reported by severity. Each alarm severity has a threshold. You can view them on the FusionInsight Manager configuration page.
+Table 2 Node monitoring metrics Metric Group
+-Table 2 Node monitoring metrics Metric Group
Metric
+Metric
Description
+Description
Default Threshold
+Default Threshold
CPU
+CPU
Host CPU Usage
+Host CPU Usage
This indicator reflects the computing and control capabilities of the current cluster in a measurement period. By observing the indicator value, you can better understand the overall resource usage of the cluster.
+This indicator reflects the computing and control capabilities of the current cluster in a measurement period. By observing the indicator value, you can better understand the overall resource usage of the cluster.
90.0%
+90.0%
Disk
+Disk
Disk Usage
+Disk Usage
Indicates the disk usage of a host.
+Indicates the disk usage of a host.
90.0%
+90.0%
Disk Inode Usage
+Disk Inode Usage
Indicates the disk inode usage in a measurement period.
+Indicates the disk inode usage in a measurement period.
80.0%
+80.0%
Memory
+Memory
Host Memory Usage
+Host Memory Usage
Indicates the average memory usage at the current time.
+Indicates the average memory usage at the current time.
90.0%
+90.0%
Host Status
+Host Status
Host File Handle Usage
+Host File Handle Usage
Indicates the usage of file handles of the host in a measurement period.
+Indicates the usage of file handles of the host in a measurement period.
80.0%
+80.0%
Host PID Usage
+Host PID Usage
Indicates the PID usage of a host.
+Indicates the PID usage of a host.
90%
+90%
Network Status
+Network Status
TCP Ephemeral Port Usage
+TCP Ephemeral Port Usage
Indicates the usage of temporary TCP ports of the host in a measurement period.
+Indicates the usage of temporary TCP ports of the host in a measurement period.
80.0%
+80.0%
Network Reading
+Network Reading
Read Packet Error Rate
+Read Packet Error Rate
Indicates the read packet error rate of the network interface on the host in a measurement period.
+Indicates the read packet error rate of the network interface on the host in a measurement period.
0.5%
+0.5%
Read Packet Dropped Rate
+Read Packet Dropped Rate
Indicates the read packet dropped rate of the network interface on the host in a measurement period.
+Indicates the read packet dropped rate of the network interface on the host in a measurement period.
0.5%
+0.5%
Read Throughput Rate
+Read Throughput Rate
Indicates the average read throughput (at MAC layer) of the network interface in a measurement period.
+Indicates the average read throughput (at MAC layer) of the network interface in a measurement period.
80%
+80%
Network Writing
+Network Writing
Write Packet Error Rate
+Write Packet Error Rate
Indicates the write packet error rate of the network interface on the host in a measurement period.
+Indicates the write packet error rate of the network interface on the host in a measurement period.
0.5%
+0.5%
Write Packet Dropped Rate
+Write Packet Dropped Rate
Indicates the write packet dropped rate of the network interface on the host in a measurement period.
+Indicates the write packet dropped rate of the network interface on the host in a measurement period.
0.5%
+0.5%
Write Throughput Rate
+Write Throughput Rate
Indicates the average write throughput (at MAC layer) of the network interface in a measurement period.
+Indicates the average write throughput (at MAC layer) of the network interface in a measurement period.
80%
+80%
Process
+Process
Uninterruptible Sleep Process
+Uninterruptible Sleep Process
Number of D state processes on the host in a measurement period
+Number of D state processes on the host in a measurement period
0
+0
omm Process Usage
+omm Process Usage
omm process usage in a measurement period
+omm process usage in a measurement period
90
+90
Table 3 Cluster service indicators Service
+Table 3 Cluster service indicators Service
Monitoring Indicator Group Name
+Monitoring Indicator Group Name
Indicator Name
+Indicator Name
Description
+Description
Default Threshold
+Default Threshold
DBService
+DBService
Database
+Database
Usage of the Number of Database Connections
+Usage of the Number of Database Connections
Indicates the usage of the number of database connections.
+Indicates the usage of the number of database connections.
90%
+90%
Disk Space Usage of the Data Directory
+Disk Space Usage of the Data Directory
Disk space usage of the data directory
+Disk space usage of the data directory
80%
+80%
Flume
+Flume
Agent
+Agent
Heap Memory Usage Calculate
+Heap Memory Usage Calculate
Indicates the Flume heap memory usage.
+Indicates the Flume heap memory usage.
95.0%
+95.0%
Flume Direct Memory Usage Statistics
+Flume Direct Memory Usage Statistics
Indicates the Flume direct memory usage.
+Indicates the Flume direct memory usage.
80.0%
+80.0%
Flume Non-heap Memory Usage
+Flume Non-heap Memory Usage
Indicates the Flume non-heap memory usage.
+Indicates the Flume non-heap memory usage.
80.0%
+80.0%
Total GC duration of Flume process
+Total GC duration of Flume process
Indicates the Flume total GC time.
+Indicates the Flume total GC time.
12000 ms
+12000 ms
HBase
+HBase
GC
+GC
GC time for old generation
+GC time for old generation
Total GC time of RegionServer
+Total GC time of RegionServer
5000 ms
+5000 ms
GC time for old generation
+GC time for old generation
Indicates he total GC time of HMaster.
+Indicates he total GC time of HMaster.
5000 ms
+5000 ms
CPU & memory
+CPU & memory
RegionServer Direct Memory Usage Statistics
+RegionServer Direct Memory Usage Statistics
Indicates theRegionServerReg direct memory usage.
+Indicates theRegionServerReg direct memory usage.
90%
+90%
RegionServer Heap Memory Usage Statistics
+RegionServer Heap Memory Usage Statistics
Indicates the RegionServer heap memory usage.
+Indicates the RegionServer heap memory usage.
90%
+90%
HMaster Direct Memory Usage
+HMaster Direct Memory Usage
Indicates the HMaster direct memory usage.
+Indicates the HMaster direct memory usage.
90%
+90%
HMaster Heap Memory Usage Statistics
+HMaster Heap Memory Usage Statistics
Indicates the HMaster heap memory usage.
+Indicates the HMaster heap memory usage.
90%
+90%
Service
+Service
Number of Online Regions of a RegionServer
+Number of Online Regions of a RegionServer
Number of regions of a RegionServer
+Number of regions of a RegionServer
2000
+2000
Region in transaction count over threshold
+Region in transaction count over threshold
Number of regions that are in the RIT state and reach the threshold duration
+Number of regions that are in the RIT state and reach the threshold duration
1
+1
Replication
+Replication
Replication sync failed times (RegionServer)
+Replication sync failed times (RegionServer)
Indicates the number of times that DR data fails to be synchronized.
+Indicates the number of times that DR data fails to be synchronized.
1
+1
Number of Log Files to Be Synchronized in the Active Cluster
+Number of Log Files to Be Synchronized in the Active Cluster
Number of log files to be synchronized in the active cluster
+Number of log files to be synchronized in the active cluster
128
+128
Number of HFiles to Be Synchronized in the Active Cluster
+Number of HFiles to Be Synchronized in the Active Cluster
Number of HFiles to be synchronized in the active cluster
+Number of HFiles to be synchronized in the active cluster
128
+128
Queue
+Queue
Compaction Queue Size
+Compaction Queue Size
Size of the Compaction queue
+Size of the Compaction queue
100
+100
HDFS
+HDFS
File and Block
+File and Block
Lost Blocks
+Lost Blocks
Indicates the number of block copies that the HDFS lacks of.
+Indicates the number of block copies that the HDFS lacks of.
0
+0
Blocks Under Replicated
+Blocks Under Replicated
Total number of blocks that need to be replicated by the NameNode
+Total number of blocks that need to be replicated by the NameNode
1000
+1000
RPC
+RPC
Average Time of Active NameNode RPC Processing
+Average Time of Active NameNode RPC Processing
Indicates the average RPC processing time.
+Indicates the average RPC processing time.
100 ms
+100 ms
Average Time of Active NameNode RPC Queuing
+Average Time of Active NameNode RPC Queuing
Indicates the average RPC queuing time.
+Indicates the average RPC queuing time.
200 ms
+200 ms
Disk
+Disk
HDFS Disk Usage
+HDFS Disk Usage
Indicates the HDFS disk usage.
+Indicates the HDFS disk usage.
80%
+80%
DataNode Disk Usage
+DataNode Disk Usage
Indicates the disk usage of DataNodes in the HDFS.
+Indicates the disk usage of DataNodes in the HDFS.
80%
+80%
Percentage of Reserved Space for Replicas of Unused Space
+Percentage of Reserved Space for Replicas of Unused Space
Indicates the percentage of the reserved disk space of all the copies to the total unused disk space of DataNodes.
+Indicates the percentage of the reserved disk space of all the copies to the total unused disk space of DataNodes.
90%
+90%
Resource
+Resource
Faulty DataNodes
+Faulty DataNodes
Indicates the number of faulty DataNodes.
+Indicates the number of faulty DataNodes.
3
+3
NameNode Non Heap Memory Usage Statistics
+NameNode Non Heap Memory Usage Statistics
Indicates the percentage of NameNode non-heap memory usage.
+Indicates the percentage of NameNode non-heap memory usage.
90%
+90%
NameNode Direct Memory Usage Statistics
+NameNode Direct Memory Usage Statistics
Indicates the percentage of direct memory used by NameNodes.
+Indicates the percentage of direct memory used by NameNodes.
90%
+90%
NameNode Heap Memory Usage Statistics
+NameNode Heap Memory Usage Statistics
Indicates the percentage of NameNode non-heap memory usage.
+Indicates the percentage of NameNode non-heap memory usage.
95%
+95%
DataNode Direct Memory Usage Statistics
+DataNode Direct Memory Usage Statistics
Indicates the percentage of direct memory used by DataNodes.
+Indicates the percentage of direct memory used by DataNodes.
90%
+90%
DataNode Heap Memory Usage Statistics
+DataNode Heap Memory Usage Statistics
DataNode heap memory usage
+DataNode heap memory usage
95%
+95%
DataNode Heap Memory Usage Statistics
+DataNode Heap Memory Usage Statistics
Indicates the percentage of DataNode non-heap memory usage.
+Indicates the percentage of DataNode non-heap memory usage.
90%
+90%
Garbage Collection
+Garbage Collection
GC Time (NameNode)/GC Time (DataNode)
+GC Time (NameNode)/GC Time (DataNode)
Indicates the Garbage collection (GC) duration of NameNodes per minute.
+Indicates the Garbage collection (GC) duration of NameNodes per minute.
12000 ms
+12000 ms
GC Time
+GC Time
Indicates the GC duration of DataNodes per minute.
+Indicates the GC duration of DataNodes per minute.
12000 ms
+12000 ms
Hive
+Hive
HQL
+HQL
Percentage of HQL Statements That Are Executed Successfully by Hive
+Percentage of HQL Statements That Are Executed Successfully by Hive
Indicates the percentage of HQL statements that are executed successfully by Hive.
+Indicates the percentage of HQL statements that are executed successfully by Hive.
90.0%
+90.0%
Background
+Background
Background Thread Usage
+Background Thread Usage
Background thread usage
+Background thread usage
90%
+90%
GC
+GC
Total GC time of MetaStore
+Total GC time of MetaStore
Indicates the total GC time of MetaStore.
+Indicates the total GC time of MetaStore.
12000 ms
+12000 ms
Total GC Time in Milliseconds
+Total GC Time in Milliseconds
Indicates the total GC time of HiveServer.
+Indicates the total GC time of HiveServer.
12000 ms
+12000 ms
Capacity
+Capacity
Percentage of HDFS Space Used by Hive to the Available Space
+Percentage of HDFS Space Used by Hive to the Available Space
Indicates the percentage of HDFS space used by Hive to the available space.
+Indicates the percentage of HDFS space used by Hive to the available space.
85.0%
+85.0%
CPU & memory
+CPU & memory
MetaStore Direct Memory Usage Statistics
+MetaStore Direct Memory Usage Statistics
MetaStore direct memory usage
+MetaStore direct memory usage
95%
+95%
MetaStore Non-Heap Memory Usage Statistics
+MetaStore Non-Heap Memory Usage Statistics
MetaStore non-heap memory usage
+MetaStore non-heap memory usage
95%
+95%
MetaStore Heap Memory Usage Statistics
+MetaStore Heap Memory Usage Statistics
MetaStore heap memory usage
+MetaStore heap memory usage
95%
+95%
HiveServer Direct Memory Usage Statistics
+HiveServer Direct Memory Usage Statistics
HiveServer direct memory usage
+HiveServer direct memory usage
95%
+95%
HiveServer Non-Heap Memory Usage Statistics
+HiveServer Non-Heap Memory Usage Statistics
HiveServer non-heap memory usage
+HiveServer non-heap memory usage
95%
+95%
HiveServer Heap Memory Usage Statistics
+HiveServer Heap Memory Usage Statistics
HiveServer heap memory usage
+HiveServer heap memory usage
95%
+95%
Session
+Session
Percentage of Sessions Connected to the HiveServer to Maximum Number of Sessions Allowed by the HiveServer
+Percentage of Sessions Connected to the HiveServer to Maximum Number of Sessions Allowed by the HiveServer
Indicates the percentage of the number of sessions connected to the HiveServer to the maximum number of sessions allowed by the HiveServer.
+Indicates the percentage of the number of sessions connected to the HiveServer to the maximum number of sessions allowed by the HiveServer.
90.0%
+90.0%
Kafka
+Kafka
Partition
+Partition
Percentage of Partitions That Are Not Completely Synchronized
+Percentage of Partitions That Are Not Completely Synchronized
Indicates the percentage of partitions that are not completely synchronized to total partitions.
+Indicates the percentage of partitions that are not completely synchronized to total partitions.
50%
+50%
Others
+Others
Unavailable Partition Percentage
+Unavailable Partition Percentage
Percentage of unavailable partitions of each Kafka topic
+Percentage of unavailable partitions of each Kafka topic
40%
+40%
User Connection Usage on Broker
+User Connection Usage on Broker
Usage of user connections on Broker
+Usage of user connections on Broker
80%
+80%
Disk
+Disk
Broker Disk Usage
+Broker Disk Usage
Indicates the disk usage of the disk where the Broker data directory is located.
+Indicates the disk usage of the disk where the Broker data directory is located.
80.0%
+80.0%
Disk I/O Rate of a Broker
+Disk I/O Rate of a Broker
I/O usage of the disk where the Broker data directory is located
+I/O usage of the disk where the Broker data directory is located
80%
+80%
Process
+Process
Broker GC Duration per Minute
+Broker GC Duration per Minute
Indicates the GC duration of the Broker process per minute.
+Indicates the GC duration of the Broker process per minute.
12000 ms
+12000 ms
Heap Memory Usage of Kafka
+Heap Memory Usage of Kafka
Indicates the Kafka heap memory usage.
+Indicates the Kafka heap memory usage.
95%
+95%
Kafka Direct Memory Usage
+Kafka Direct Memory Usage
Indicates the Kafka direct memory usage.
+Indicates the Kafka direct memory usage.
95%
+95%
Loader
+Loader
Memory
+Memory
Heap Memory Usage Calculate
+Heap Memory Usage Calculate
Indicates the Loader heap memory usage.
+Indicates the Loader heap memory usage.
95%
+95%
Direct Memory Usage of Loader
+Direct Memory Usage of Loader
Indicates the Loader direct memory usage.
+Indicates the Loader direct memory usage.
80.0%
+80.0%
Non-heap Memory Usage of Loader
+Non-heap Memory Usage of Loader
Indicates the Loader non-heap memory usage.
+Indicates the Loader non-heap memory usage.
80%
+80%
GC
+GC
Total GC time of Loader
+Total GC time of Loader
Indicates the total GC time of Loader.
+Indicates the total GC time of Loader.
12000 ms
+12000 ms
MapReduce
+MapReduce
Garbage Collection
+Garbage Collection
GC Time
+GC Time
Indicates the GC time.
+Indicates the GC time.
12000 ms
+12000 ms
Resource
+Resource
JobHistoryServer Direct Memory Usage Statistics
+JobHistoryServer Direct Memory Usage Statistics
Indicates the JobHistoryServer direct memory usage.
+Indicates the JobHistoryServer direct memory usage.
90%
+90%
JobHistoryServer Non Heap Memory Usage Statistics
+JobHistoryServer Non Heap Memory Usage Statistics
Indicates the JobHistoryServer non-heap memory usage.
+Indicates the JobHistoryServer non-heap memory usage.
90%
+90%
JobHistoryServer Heap Memory Usage Statistics
+JobHistoryServer Heap Memory Usage Statistics
Indicates the JobHistoryServer non-heap memory usage.
+Indicates the JobHistoryServer non-heap memory usage.
95%
+95%
Oozie
+Oozie
Memory
+Memory
Heap Memory Usage Calculate
+Heap Memory Usage Calculate
Indicates the Oozie heap memory usage.
+Indicates the Oozie heap memory usage.
95.0%
+95.0%
Oozie Direct Memory Usage
+Oozie Direct Memory Usage
Indicates the Oozie direct memory usage.
+Indicates the Oozie direct memory usage.
80.0%
+80.0%
Oozie Non-heap Memory Usage
+Oozie Non-heap Memory Usage
Indicates the Oozie non-heap memory usage.
+Indicates the Oozie non-heap memory usage.
80%
+80%
GC
+GC
Total GC duration of Oozie
+Total GC duration of Oozie
Indicates the Oozie total GC time.
+Indicates the Oozie total GC time.
12000 ms
+12000 ms
Spark2x
+Spark2x
Memory
+Memory
JDBCServer2x Heap Memory Usage Statistics
+JDBCServer2x Heap Memory Usage Statistics
JDBCServer2x heap memory usage
+JDBCServer2x heap memory usage
95%
+95%
JDBCServer2x Direct Memory Usage Statistics
+JDBCServer2x Direct Memory Usage Statistics
JDBCServer2x direct memory usage
+JDBCServer2x direct memory usage
95%
+95%
JDBCServer2x Non-Heap Memory Usage Statistics
+JDBCServer2x Non-Heap Memory Usage Statistics
JDBCServer2x non-heap memory usage
+JDBCServer2x non-heap memory usage
95%
+95%
JobHistory2x Direct Memory Usage Statistics
+JobHistory2x Direct Memory Usage Statistics
JobHistory2x direct memory usage
+JobHistory2x direct memory usage
95%
+95%
JobHistory2x Non-Heap Memory Usage Statistics
+JobHistory2x Non-Heap Memory Usage Statistics
JobHistory2x non-heap memory usage
+JobHistory2x non-heap memory usage
95%
+95%
JobHistory2x Heap Memory Usage Statistics
+JobHistory2x Heap Memory Usage Statistics
JobHistory2x heap memory usage
+JobHistory2x heap memory usage
95%
+95%
IndexServer2x Direct Memory Usage Statistics
+IndexServer2x Direct Memory Usage Statistics
IndexServer2x direct memory usage
+IndexServer2x direct memory usage
95%
+95%
IndexServer2x Heap Memory Usage Statistics
+IndexServer2x Heap Memory Usage Statistics
IndexServer2x heap memory usage
+IndexServer2x heap memory usage
95%
+95%
IndexServer2x Non-Heap Memory Usage Statistics
+IndexServer2x Non-Heap Memory Usage Statistics
IndexServer2x non-heap memory usage
+IndexServer2x non-heap memory usage
95%
+95%
GC Count
+GC Count
Full GC Number of JDBCServer2x
+Full GC Number of JDBCServer2x
Total GC number of JDBCServer2x
+Total GC number of JDBCServer2x
12
+12
Full GC Number of JobHistory2x
+Full GC Number of JobHistory2x
Total GC number of JobHistory2x
+Total GC number of JobHistory2x
12
+12
Full GC Number of IndexServer2x
+Full GC Number of IndexServer2x
Total GC number of IndexServer2x
+Total GC number of IndexServer2x
12
+12
GC Time
+GC Time
Total GC Time in Milliseconds
+Total GC Time in Milliseconds
Total GC time of JDBCServer2x
+Total GC time of JDBCServer2x
12000 ms
+12000 ms
Total GC Time in Milliseconds
+Total GC Time in Milliseconds
Total GC time of JobHistory2x
+Total GC time of JobHistory2x
12000 ms
+12000 ms
Total GC Time in Milliseconds
+Total GC Time in Milliseconds
Total GC time of IndexServer2x
+Total GC time of IndexServer2x
12000 ms
+12000 ms
Storm
+Storm
Cluster
+Cluster
Number of Available Supervisors
+Number of Available Supervisors
Indicates the number of available Supervisor processes in the cluster in a measurement period.
+Indicates the number of available Supervisor processes in the cluster in a measurement period.
1
+1
Slot Usage
+Slot Usage
Indicates the slot usage in the cluster in a measurement period.
+Indicates the slot usage in the cluster in a measurement period.
80.0%
+80.0%
Nimbus
+Nimbus
Heap Memory Usage Calculate
+Heap Memory Usage Calculate
Indicates the Nimbus heap memory usage.
+Indicates the Nimbus heap memory usage.
80%
+80%
Yarn
+Yarn
Resources
+Resources
NodeManager Direct Memory Usage Statistics
+NodeManager Direct Memory Usage Statistics
Indicates the percentage of direct memory used by NodeManagers.
+Indicates the percentage of direct memory used by NodeManagers.
90%
+90%
NodeManager Heap Memory Usage Statistics
+NodeManager Heap Memory Usage Statistics
Indicates the percentage of NodeManager heap memory usage.
+Indicates the percentage of NodeManager heap memory usage.
95%
+95%
NodeManager Non Heap Memory Usage Statistics
+NodeManager Non Heap Memory Usage Statistics
Indicates the percentage of NodeManager non-heap memory usage.
+Indicates the percentage of NodeManager non-heap memory usage.
90%
+90%
ResourceManager Direct Memory Usage Statistics
+ResourceManager Direct Memory Usage Statistics
Indicates the Kafka direct memory usage.
+Indicates the Kafka direct memory usage.
90%
+90%
ResourceManager Heap Memory Usage Statistics
+ResourceManager Heap Memory Usage Statistics
Indicates the ResourceManager heap memory usage.
+Indicates the ResourceManager heap memory usage.
95%
+95%
ResourceManager Non Heap Memory Usage Statistics
+ResourceManager Non Heap Memory Usage Statistics
Indicates the ResourceManager non-heap memory usage.
+Indicates the ResourceManager non-heap memory usage.
90%
+90%
Garbage collection
+Garbage collection
GC Time
+GC Time
Indicates the GC duration of NodeManager per minute.
+Indicates the GC duration of NodeManager per minute.
12000 ms
+12000 ms
GC Time
+GC Time
Indicates the GC duration of ResourceManager per minute.
+Indicates the GC duration of ResourceManager per minute.
12000 ms
+12000 ms
Others
+Others
Failed Applications of root queue
+Failed Applications of root queue
Number of failed tasks in the root queue
+Number of failed tasks in the root queue
50
+50
Terminated Applications of root queue
+Terminated Applications of root queue
Number of killed tasks in the root queue
+Number of killed tasks in the root queue
50
+50
CPU & memory
+CPU & memory
Pending Memory
+Pending Memory
Pending memory capacity
+Pending memory capacity
83886080MB
+83886080MB
Application
+Application
Pending Applications
+Pending Applications
Pending tasks
+Pending tasks
60
+60
ZooKeeper
+ZooKeeper
Connection
+Connection
ZooKeeper Connections Usage
+ZooKeeper Connections Usage
Indicates the percentage of the used connections to the total connections of ZooKeeper.
+Indicates the percentage of the used connections to the total connections of ZooKeeper.
80%
+80%
CPU & memory
+CPU & memory
Directmemory Usage Calculate
+Directmemory Usage Calculate
Indicates the ZooKeeper heap memory usage.
+Indicates the ZooKeeper heap memory usage.
95%
+95%
Heap Memory Usage Calculate
+Heap Memory Usage Calculate
Indicates the ZooKeeper direct memory usage.
+Indicates the ZooKeeper direct memory usage.
80%
+80%
GC
+GC
ZooKeeper GC Duration per Minute
+ZooKeeper GC Duration per Minute
Indicates the GC time of ZooKeeper every minute.
+Indicates the GC time of ZooKeeper every minute.
12000 ms
+12000 ms
meta
+meta
OBS data write operation
+OBS data write operation
Success Rate for Calling the OBS Write API
+Success Rate for Calling the OBS Write API
Success rate for calling the OBS data read API
+Success rate for calling the OBS data read API
99.0%
+99.0%
OBS Meta data Operations
+OBS Meta data Operations
Average Time for Calling the OBS Metadata API
+Average Time for Calling the OBS Metadata API
Average time for calling the OBS metadata API
+Average time for calling the OBS metadata API
500ms
+500ms
Success Rate for Calling the OBS Metadata API
+Success Rate for Calling the OBS Metadata API
Success rate for calling the OBS metadata API
+Success rate for calling the OBS metadata API
99.0%
+99.0%
OBS data read operation
+OBS data read operation
Success Rate for Calling the OBS Data Read API
+Success Rate for Calling the OBS Data Read API
Success rate for calling the OBS data read API
+Success rate for calling the OBS data read API
99.0%
+99.0%
Ranger
+Ranger
GC
+GC
UserSync GC Duration
+UserSync GC Duration
UserSync garbage collection (GC) duration
+UserSync garbage collection (GC) duration
12000 ms
+12000 ms
RangerAdmin GC Duration
+RangerAdmin GC Duration
RangerAdmin GC duration
+RangerAdmin GC duration
12000 ms
+12000 ms
TagSync GC Duration
+TagSync GC Duration
TagSync GC duration
+TagSync GC duration
12000 ms
+12000 ms
CPU & memory
+CPU & memory
UserSync Non-Heap Memory Usage
+UserSync Non-Heap Memory Usage
UserSync non-heap memory usage
+UserSync non-heap memory usage
80.0%
+80.0%
UserSync Direct Memory Usage
+UserSync Direct Memory Usage
UserSync direct memory usage
+UserSync direct memory usage
80.0%
+80.0%
UserSync Heap Memory Usage
+UserSync Heap Memory Usage
UserSync heap memory usage
+UserSync heap memory usage
95.0%
+95.0%
RangerAdmin Non-Heap Memory Usage
+RangerAdmin Non-Heap Memory Usage
RangerAdmin non-heap memory usage
+RangerAdmin non-heap memory usage
80.0%
+80.0%
RangerAdmin Heap Memory Usage
+RangerAdmin Heap Memory Usage
RangerAdmin heap memory usage
+RangerAdmin heap memory usage
95.0%
+95.0%
RangerAdmin Direct Memory Usage
+RangerAdmin Direct Memory Usage
RangerAdmin direct memory usage
+RangerAdmin direct memory usage
80.0%
+80.0%
TagSync Direct Memory Usage
+TagSync Direct Memory Usage
TagSync direct memory usage
+TagSync direct memory usage
80.0%
+80.0%
TagSync Non-Heap Memory Usage
+TagSync Non-Heap Memory Usage
TagSync non-heap memory usage
+TagSync non-heap memory usage
80.0%
+80.0%
TagSync Heap Memory Usage
+TagSync Heap Memory Usage
TagSync heap memory usage
+TagSync heap memory usage
95.0%
+95.0%
ClickHouse
+ClickHouse
Cluster Quota
+Cluster Quota
Clickhouse service quantity quota usage in ZooKeeper
+Clickhouse service quantity quota usage in ZooKeeper
Quota of the ZooKeeper nodes used by a ClickHouse service
+Quota of the ZooKeeper nodes used by a ClickHouse service
90%
+90%
Capacity quota usage of the Clickhouse service in ZooKeeper
+Capacity quota usage of the Clickhouse service in ZooKeeper
Capacity quota of ZooKeeper directory used by the ClickHouse service
+Capacity quota of ZooKeeper directory used by the ClickHouse service
90%
+90%
Log Download
-Scenario
MRS Manager allows you to batch export logs generated on all instances of each service.
Procedure
- Log in to MRS Manager.
- Choose O&M > Log > Download.
- Select a log download range:
- Service: Click
and select a service. - Host: Enter the IP address of the host where the service is deployed. You can also click
to select the required host. - Click
in the upper right corner and configure Start Time and End Time.
Procedure
- Log in to MRS Manager.
- Choose O&M > Log > Download.
- Select a log download range:
- Service: Click and select a service.
- Host: Enter the IP address of the host where the service is deployed. You can also click to select the required host.
- Max. Concurrent Nodes: Set the maximum number of concurrent nodes for log collection as you need. MRS 3.3.0 or later supports this parameter.
- Click in the upper right corner and configure Start Time and End Time.
- Service: Click
- Click Download.
The downloaded log package contains the topology information of the start time and end time, helping you quickly find the log you need.
The topology file is named in the format of topo_<Topology structure change time>.txt. The file contains the node IP address, host name, and service instances that reside on the node. (OMS nodes are identified by Manager:Manager.)
Example:
diff --git a/docs/mrs/umn/admin_guide_000101.html b/docs/mrs/umn/admin_guide_000101.html index b14ab088..d6cc8b5f 100644 --- a/docs/mrs/umn/admin_guide_000101.html +++ b/docs/mrs/umn/admin_guide_000101.html @@ -14,7 +14,7 @@Cluster
Indicates the cluster for which you want to create a tenant.
+Indicates the cluster for which you want to create a tenant. (This parameter is unavailable for clusters of MRS 3.3.0 or later.)
Name
@@ -25,7 +25,9 @@Tenant Resource Type
Specifies whether the tenant is a leaf tenant.
-- When Leaf Tenant Resource is selected, the current tenant is a leaf tenant and no sub-tenant can be added.
- When Non-leaf Tenant Resource is selected, the current tenant is not a leaf tenant and sub-tenants can be added to the current tenant.
- When Leaf Tenant Resource is selected, the current tenant is a leaf tenant and no sub-tenant can be added.
- When Non-leaf Tenant Resource is selected, the current tenant is not a leaf tenant and sub-tenants can be added to the current tenant.NOTE:+
MRS 3.2.0 or later: If you select ClickHouse for Service, this parameter can only be set to Leaf Tenant.
+
Computing Resource
@@ -108,10 +110,12 @@- Check whether the current tenant needs to be associated with resources of other services.
- Click Associate Service to configure other service resources used by the current tenant, and click OK.
- Set Service to HBase and Association Type to Exclusive or Shared.+
- Exclusive indicates that the service resources are used by the tenant exclusively and cannot be associated with other tenants.
- Shared indicates that the service resources can be shared with other tenants.
- MRS 3.2.0 or later: Set Service to ClickHouse.
- Association Type: When Service is set to ClickHouse, Association Type can only be set to Shared. MRS 3.3.0 and later versions support Exclusive and Shared.
- Associate Logical Cluster: If the logical cluster function is not enabled for ClickHouse, default_cluster is selected by default. If the function is enabled, select the logical cluster to which you want to associate.
- CPU Priority: The CPU priority ranges from –20 to 19. This value is associated with the NICE value of the OS. A smaller value indicates a higher CPU priority.
- Memory: The maximum value of this parameter is 100, in percentage. For example, if this parameter is set to 80, the total memory that can be used by the current tenant is calculated as follows: Available memory x 80%.
- Concurrency: The maximum number of concurrent resources available for all the users bound to the tenant. This parameter is required for clusters of MRS 3.3.0 or later.
- - Only HBase can be associated with a new tenant. However, HDFS, HBase, and Yarn can be associated with existing tenants.
- To associate an existing tenant with service resources, click the target tenant in the tenant list, switch to the Service Associations page, and click Associate Service to configure resources to be associated with the tenant.
- To disassociate an existing tenant from service resources, click the target tenant in the tenant list, switch to the Service Associations page, and click Delete in the Operation column. In the displayed dialog box, select I have read the information and understand the impact and click OK.
- Click OK. Wait until the system displays a message indicating that the tenant is successfully created.
+- Click OK. Wait until the system displays a message indicating that the tenant is successfully created.
For clusters of MRS 3.3.0 or later, click
+
to export basic information about all tenants in the current cluster.diff --git a/docs/mrs/umn/admin_guide_000109.html b/docs/mrs/umn/admin_guide_000109.html index bc55ecd8..8820e3e7 100644 --- a/docs/mrs/umn/admin_guide_000109.html +++ b/docs/mrs/umn/admin_guide_000109.html @@ -4,7 +4,7 @@-Scenario
In a cluster, you can logically group Yarn NodeManagers into Yarn resource pools. Each NodeManager belongs to only one resource pool. You can create a custom resource pool on MRS Manager and add the hosts that have not been added to any custom resource pools to this resource pool so that specified queues can use the computing resources provided by these hosts.
The system contains a default resource pool by default. All NodeManagers that are not added to custom resource pools belong to this resource pool.
Procedure
- Log in to MRS Manager.
- Choose .
- Click Add Resource Pool.
- Set resource pool attributes.
- Cluster: Select the cluster to which the resource pool is to be added.
- Name: Enter the name of the resource pool. The name contains 1 to 50 characters, including digits, letters, and underscores (_), and cannot start with an underscore (_).
- Resource Label: Enter the resource label of the resource pool. The value can contain 1 to 50 characters, including digits, letters, underscores (_), and hyphens (-), and must start with a digit or letter.
- Resource: In the Available Hosts area, select specified hosts and click
to add the hosts to the Selected Hosts area. Only hosts in the cluster can be selected. The host list in the resource pool can be left blank. You can filter hosts by host name, number of CPU cores, memory, operating system, or platform type based on service requirements.
+Procedure
- Log in to MRS Manager.
- Choose .
- Click Add Resource Pool.
- Set resource pool attributes.
- Cluster: Select the cluster to which the resource pool is to be added. (This parameter is unavailable for clusters of MRS 3.3.0 or later.)
- Name: Enter the name of the resource pool. The name contains 1 to 50 characters, including digits, letters, and underscores (_), and cannot start with an underscore (_).
- Resource Label: Enter the resource label of the resource pool. The value can contain 1 to 50 characters, including digits, letters, underscores (_), and hyphens (-), and must start with a digit or letter.
- Resource: In the Available Hosts area, select specified hosts and click to add the hosts to the Selected Hosts area. Only hosts in the cluster can be selected. The host list in the resource pool can be left blank.
You can filter hosts by host name, number of CPU cores, memory, operating system, or platform type based on service requirements.
- Click OK.
After the resource pool is created, you can view its name, members, and mode in the resource pool list. Hosts that are added to the custom resource pool are no longer members of the default resource pool.
diff --git a/docs/mrs/umn/admin_guide_000110.html b/docs/mrs/umn/admin_guide_000110.html index 838ec1dc..8003c23b 100644 --- a/docs/mrs/umn/admin_guide_000110.html +++ b/docs/mrs/umn/admin_guide_000110.html @@ -3,7 +3,7 @@Modifying a Resource Pool
-Scenario
When hosts in a resource pool need to be adjusted based on service requirements, you can modify members in the resource pool on MRS Manager.
diff --git a/docs/mrs/umn/admin_guide_000112.html b/docs/mrs/umn/admin_guide_000112.html index fa3de5a2..8ff6bcab 100644 --- a/docs/mrs/umn/admin_guide_000112.html +++ b/docs/mrs/umn/admin_guide_000112.html @@ -55,6 +55,18 @@Procedure
- Log in to MRS Manager.
- Choose .
- Locate the row that contains the specified resource pool, and click Edit in the Operation column.
- In the Resource area, modify hosts.
- Adding hosts: Select desired hosts in Available Hosts and click
to add them to the resource pool. - Deleting hosts: Select desired hosts in Selected Hosts and click
to remove them from the resource pool. The host list in the resource pool can be left blank.
Procedure
- Log in to MRS Manager.
- Choose .
- Locate the row that contains the specified resource pool, and click Edit in the Operation column (for MRS 3.3.0 or later, click Modify).
- In the Resource area, modify hosts.
- Adding hosts: Select desired hosts in Available Hosts and click to add them to the resource pool.
- Deleting hosts: Select desired hosts in Selected Hosts and click to remove them from the resource pool. The host list in the resource pool can be left blank.
- Adding hosts: Select desired hosts in Available Hosts and click
- Click OK.
- Adding hosts: Select desired hosts in Available Hosts and click
Indicates that tasks are executed on a node with a specified resource label.
Cross-Pool Scheduling
+Indicates whether containers in the current queue support cross-pool scheduling. This parameter is available for clusters of MRS 3.3.0 or later.
+This function cannot be enabled for the default queue.
+Cross-Pool AM Scheduling
+Indicates whether ApplicationMasters in the current queue support cross-pool scheduling. This parameter is available for clusters of MRS 3.3.0 or later.
+This function cannot be enabled for the default queue.
+Active
- ACTIVE: indicates that the current queue can receive and execute tasks.
- INACTIVE: indicates that the current queue can receive but cannot execute tasks. Tasks submitted to the queue are suspended.
Procedure
- Add a policy.
- On MRS Manager, choose Tenant Resources.
- Choose Dynamic Resource Plan.
- Click the Global User Policy tab.-
defaults(default setting) indicates that the policy specified for defaults is used if a user does not have a global policy. The default policy cannot be deleted.
- Click Create Global User Policy. In the displayed dialog box, set the following parameters:
- Cluster: Select the target cluster.
- Username: indicates the user for whom resource scheduling is controlled. Enter an existing username in the current cluster.
- Max Running Apps: indicates the maximum number of tasks that the user can run in the current cluster.
- Max Pending Apps: indicates the maximum number of tasks that the user can suspend in the current cluster.
- Default Queue: indicates the queue of the user. Enter the name of an existing queue in the current cluster.
- Click Create Global User Policy. In the displayed dialog box, set the following parameters:
- Cluster: Select the target cluster. (This parameter is unavailable for clusters of MRS 3.3.0 or later.)
- Username: indicates the user for whom resource scheduling is controlled. Enter an existing username in the current cluster.
- Max Running Apps: indicates the maximum number of tasks that the user can run in the current cluster.
- Max Pending Apps: indicates the maximum number of tasks that the user can suspend in the current cluster.
- Default Queue: indicates the queue of the user. Enter the name of an existing queue in the current cluster.
- Modify a policy.
- On MRS Manager, choose Tenant Resources.
- Choose Dynamic Resource Plan.
- Click the Global User Policy tab.
- In the row that contains the desired user policy, click Modify in the Operation column.
- In the displayed dialog box, modify parameters and click OK.
Prerequisites
- A tenant name has been planned based on service requirements. The name cannot be the same as that of a role, HDFS directory, or Yarn queue that exists in the current cluster.
- Resources to be allocated to the current tenant have been planned to ensure that the sum of resources of direct sub-tenants at each level does not exceed the resources of the current tenant.
Procedure
- Log in to MRS Manager and choose Tenant Resources.
- Click
. On the page that is displayed, configure tenant attributes according to Table 1.-
diff --git a/docs/mrs/umn/admin_guide_000234.html b/docs/mrs/umn/admin_guide_000234.html index 95ed73ad..c264bf01 100644 --- a/docs/mrs/umn/admin_guide_000234.html +++ b/docs/mrs/umn/admin_guide_000234.html @@ -12,7 +12,7 @@Table 1 Tenant parameters Parameter
+@@ -66,6 +74,14 @@Table 1 Tenant parameters Parameter
Description
+Description
Cluster
+Cluster
Indicates the cluster for which you want to create a tenant.
+Indicates the cluster for which you want to create a tenant. (This parameter is unavailable for clusters of MRS 3.3.0 or later.)
Name
+Name
- Indicates the name of the current tenant. The value consists of 3 to 50 characters, including digits, letters, and underscores (_).
- Plan a tenant name based on service requirements. The name cannot be the same as that of a role, HDFS directory, or Yarn queue that exists in the current cluster.
- Indicates the name of the current tenant. The value consists of 3 to 50 characters, including digits, letters, and underscores (_).
- Plan a tenant name based on service requirements. The name cannot be the same as that of a role, HDFS directory, or Yarn queue that exists in the current cluster.
Tenant Resource Type
+Tenant Resource Type
Specifies whether the tenant is a leaf tenant.
-- When Leaf Tenant Resource is selected, the current tenant is a leaf tenant and no sub-tenant can be added.
- When Non-leaf Tenant Resource is selected, the current tenant is not a leaf tenant and sub-tenants can be added to the current tenant.
Specifies whether the tenant is a leaf tenant.
+- When Leaf Tenant Resource is selected, the current tenant is a leaf tenant and no sub-tenant can be added.
- When Non-leaf Tenant Resource is selected, the current tenant is not a leaf tenant and sub-tenants can be added to the current tenant.NOTE:+
MRS 3.2.0 or later: If you select ClickHouse for Service, this parameter can only be set to Leaf Tenant.
+
Computing Resource
+Computing Resource
Specifies the dynamic computing resources for the current tenant.
-- When Yarn is selected, the system automatically creates a queue in Yarn and the queue is named the same as the tenant name.
- A leaf tenant can directly submit jobs to the queue.
- A non-leaf tenant cannot directly submit jobs to the queue. However, Yarn adds an extra queue (hidden) named default for the non-leaf tenant to record the remaining resource capacity of the tenant. Actual jobs do not run in this queue.
- If Yarn is not selected, the system does not automatically create a queue.
Specifies the dynamic computing resources for the current tenant.
+- When Yarn is selected, the system automatically creates a queue in Yarn and the queue is named the same as the tenant name.
- A leaf tenant can directly submit jobs to the queue.
- A non-leaf tenant cannot directly submit jobs to the queue. However, Yarn adds an extra queue (hidden) named default for the non-leaf tenant to record the remaining resource capacity of the tenant. Actual jobs do not run in this queue.
- If Yarn is not selected, the system does not automatically create a queue.
Configuration Mode
+Configuration Mode
Indicates the configuration mode of computing resource parameters.
-- If you select Basic, you only need to set Default Resource Pool Capacity (%).
- If you select Advanced, you can manually configure the resource allocation weight and the minimum, maximum, and reserved resources of the tenant.
Indicates the configuration mode of computing resource parameters.
+- If you select Basic, you only need to set Default Resource Pool Capacity (%).
- If you select Advanced, you can manually configure the resource allocation weight and the minimum, maximum, and reserved resources of the tenant.
Default Resource Pool Capacity (%)
+Default Resource Pool Capacity (%)
Indicates the percentage of computing resources used by the current tenant in the default resource pool. The value ranges from 0 to 100%.
+Indicates the percentage of computing resources used by the current tenant in the default resource pool. The value ranges from 0 to 100%.
Weight
+Weight
Indicates the resource allocation weight. The value ranges from 0 to 100.
+Indicates the resource allocation weight. The value ranges from 0 to 100.
Minimum Resource
+Minimum Resource
Indicates the resources guaranteed for the tenant (preemption supported). The value can be a percentage or an absolute value of the parent tenant's resources. When a tenant has a light workload, the resources of the tenant are automatically allocated to other tenants. When the available tenant resources are less than the value of Minimum Resource, the tenant can preempt the resources that have been lent to other tenants.
+Indicates the resources guaranteed for the tenant (preemption supported). The value can be a percentage or an absolute value of the parent tenant's resources. When a tenant has a light workload, the resources of the tenant are automatically allocated to other tenants. When the available tenant resources are less than the value of Minimum Resource, the tenant can preempt the resources that have been lent to other tenants.
Maximum Resource
+Maximum Resource
Indicates the maximum resources that can be used by the tenant. The tenant cannot obtain more resources than the value configured. The value can be a percentage or an absolute value of the parent tenant's resources.
+Indicates the maximum resources that can be used by the tenant. The tenant cannot obtain more resources than the value configured. The value can be a percentage or an absolute value of the parent tenant's resources.
Reserved Resource
+Reserved Resource
Indicates the resources reserved for the tenant. The reserved resources cannot be used by other tenants even if no job is running in the current tenant resources. The value can be a percentage or an absolute value of the parent tenant's resources.
+Indicates the resources reserved for the tenant. The reserved resources cannot be used by other tenants even if no job is running in the current tenant resources. The value can be a percentage or an absolute value of the parent tenant's resources.
Storage Resource
+Storage Resource
Specifies storage resources for the current tenant.
-- When HDFS is selected, the system automatically allocates storage resources.
- When HDFS is not selected, the system does not automatically allocate storage resources.
Specifies storage resources for the current tenant.
+- When HDFS is selected, the system automatically allocates storage resources.
- When HDFS is not selected, the system does not automatically allocate storage resources.
Quota
+Quota
Indicates the quota for files and directories.
+Indicates the quota for files and directories.
Space Quota
+Space Quota
Indicates the quota for the HDFS storage space used by the current tenant.
-- If the unit is set to MB, the value ranges from 1 to 8796093022208. If the unit is set to GB, the value ranges from 1 to 8589934592.
- This parameter indicates the maximum HDFS storage space that can be used by the tenant, but not the actual space used.
- If its value is greater than the size of the HDFS physical disk, the maximum space available is the full space of the HDFS physical disk.
Indicates the quota for the HDFS storage space used by the current tenant.
+- If the unit is set to MB, the value ranges from 1 to 8796093022208. If the unit is set to GB, the value ranges from 1 to 8589934592.
- This parameter indicates the maximum HDFS storage space that can be used by the tenant, but not the actual space used.
- If its value is greater than the size of the HDFS physical disk, the maximum space available is the full space of the HDFS physical disk.
Storage Path
+Storage Path
Indicates the HDFS storage directory for the tenant.
-- The system automatically creates a folder named after the tenant name in the /tenant directory by default. For example, the default HDFS storage directory for tenant ta1 is /tenant/ta1.
- When a tenant is created for the first time, the system creates the /tenant directory in the HDFS root directory. The storage path is customizable.
Indicates the HDFS storage directory for the tenant.
+- The system automatically creates a folder named after the tenant name in the /tenant directory by default. For example, the default HDFS storage directory for tenant ta1 is /tenant/ta1.
- When a tenant is created for the first time, the system creates the /tenant directory in the HDFS root directory. The storage path is customizable.
Service
+Service
Specifies whether to associate resources of other services. For details, see 4.
+Specifies whether to associate resources of other services. For details, see 4.
Description
+Description
Indicates the description of the current tenant.
+Indicates the description of the current tenant.
-Scenario
In a cluster, you can logically group Yarn NodeManagers into Yarn resource pools. Each NodeManager belongs to only one resource pool. You can create a custom resource pool on MRS Manager and add the hosts that have not been added to any custom resource pools to this resource pool so that specified queues can use the computing resources provided by these hosts.
The system contains a default resource pool by default. All NodeManagers that are not added to custom resource pools belong to this resource pool.
Procedure
- Log in to MRS Manager.
- Choose .
- Click Add Resource Pool.
- Set resource pool attributes.
- Cluster: Select the cluster to which the resource pool is to be added.
- Name: Enter the name of the resource pool. The name contains 1 to 50 characters, including digits, letters, and underscores (_), and cannot start with an underscore (_).
- Resource Label: Enter the resource label of the resource pool. The value can contain 1 to 50 characters, including digits, letters, underscores (_), and hyphens (-), and must start with a digit or letter.
- Resource: In the Available Hosts area, select specified hosts and click
to add the hosts to the Selected Hosts area. Only hosts in the cluster can be selected. The host list in the resource pool can be left blank. You can filter hosts by host name, number of CPU cores, memory, operating system, or platform type based on service requirements.
+Procedure
- Log in to MRS Manager.
- Choose .
- Click Add Resource Pool.
- Set resource pool attributes.
- Cluster: Select the cluster to which the resource pool is to be added. (This parameter is unavailable for clusters of MRS 3.3.0 or later.)
- Name: Enter the name of the resource pool. The name contains 1 to 50 characters, including digits, letters, and underscores (_), and cannot start with an underscore (_).
- Resource Label: Enter the resource label of the resource pool. The value can contain 1 to 50 characters, including digits, letters, underscores (_), and hyphens (-), and must start with a digit or letter.
- Resource: In the Available Hosts area, select specified hosts and click to add the hosts to the Selected Hosts area. Only hosts in the cluster can be selected. The host list in the resource pool can be left blank.
You can filter hosts by host name, number of CPU cores, memory, operating system, or platform type based on service requirements.
- Click OK.
After the resource pool is created, you can view its name, members, and mode in the resource pool list. Hosts that are added to the custom resource pool are no longer members of the default resource pool.
diff --git a/docs/mrs/umn/admin_guide_000128.html b/docs/mrs/umn/admin_guide_000128.html index 760f2f9e..9b04b66c 100644 --- a/docs/mrs/umn/admin_guide_000128.html +++ b/docs/mrs/umn/admin_guide_000128.html @@ -3,7 +3,7 @@Modifying a Resource Pool
-Scenario
When hosts in a resource pool need to be adjusted based on service requirements, you can modify members in the resource pool on MRS Manager.
diff --git a/docs/mrs/umn/admin_guide_000133.html b/docs/mrs/umn/admin_guide_000133.html index 510b8b37..dc0fbc5f 100644 --- a/docs/mrs/umn/admin_guide_000133.html +++ b/docs/mrs/umn/admin_guide_000133.html @@ -5,11 +5,42 @@Procedure
- Log in to MRS Manager.
- Choose .
- Locate the row that contains the specified resource pool, and click Edit in the Operation column.
- In the Resource area, modify hosts.
- Adding hosts: Select desired hosts in Available Hosts and click
to add them to the resource pool. - Deleting hosts: Select desired hosts in Selected Hosts and click
to remove them from the resource pool. The host list in the resource pool can be left blank.
Procedure
- Log in to MRS Manager.
- Choose .
- Locate the row that contains the specified resource pool, and click Edit in the Operation column (for MRS 3.3.0 or later, click Modify).
- In the Resource area, modify hosts.
- Adding hosts: Select desired hosts in Available Hosts and click to add them to the resource pool.
- Deleting hosts: Select desired hosts in Selected Hosts and click to remove them from the resource pool. The host list in the resource pool can be left blank.
- Adding hosts: Select desired hosts in Available Hosts and click
- Click OK.
- Adding hosts: Select desired hosts in Available Hosts and click
-Prerequisites
- The network connectivity of the cluster is proper and secure, and the YARN service status is normal.
- During scheduler switching, tenants cannot be added, deleted, or modified. In addition, services cannot be started or stopped.
Impact on the System
- Because the ResourceManager is restarted during scheduler switching, submitting jobs to YARN will fail at that time.
- During scheduler switching, tasks in a job being executed on YARN will continue, but new tasks cannot be started.
- After scheduler switching is complete, jobs executed on YARN may fail, causing service interruptions.
- After scheduler switching is complete, parameters of the Superior scheduler are used for tenant management.
- After scheduler switching is complete, tenant queues whose capacity is 0 in the Capacity scheduler cannot be allocated resources in the Superior scheduler. As a result, jobs submitted to these tenant queues fail to be executed. Therefore, you are advised not to set the capacity of a tenant queue to 0 in the Capacity scheduler.
- After scheduler switching is complete, you cannot add or delete resource pools, YARN node labels, or tenants during the observation period. If such an operation is performed, the scheduler cannot be rolled back to the Capacity scheduler.-
- The recommended observation period for scheduler switching is one week. If resource pools, YARN node labels, or tenants are added or deleted during this period, the observation period ends immediately.
- The scheduler rollback may cause the loss of partial or all YARN job information.
Switching Between the Capacity Scheduler and Superior Scheduler (Available for Clusters of MRS 3.3.0 or Later)
This function is only available for clusters of MRS 3.3.0 or later.
+Constraints-- This operation is available for only the scenario where a cluster is newly provisioned and the scheduler needs to be switched.
- During the scheduler switchover, do not perform any operation on the cluster. Otherwise, the operation may fail due to database modification.
+Switching from the Capacity Scheduler to the Superior Scheduler
- Modify YARN service parameters and ensure that the YARN service status is normal.
- Log in to MRS Manager as an administrator.
- Log in to MRS Manager and choose Cluster > Services > Yarn. Click Configurations then All Configurations, search for yarn.resourcemanager.webapp.pagination.enable, and check whether the value is true.
- If yes, go to 1.c.
- If no, set the parameter to true and click Save to save the configuration. On the Dashboard tab page of YARN, choose More > Restart Service, verify the identity, and click OK. After the service is restarted, go to 1.c.
Impact on the system+- Because the ResourceManager is restarted during scheduler switching, submitting jobs to Yarn will fail at that time.
- After the scheduler is switched, the parameters of the scheduler that takes over the workload are used.
Procedure
+- Log in to FusionInsight Manager. Choose Cluster > Services > Yarn and check whether the Yarn service status is normal. If the service is abnormal, restore the service.
- Log in to the active management node as user omm.
- Switch the scheduler.
- Run the following command to switch from the Capacity scheduler to the Superior scheduler:
sh ${BIGDATA_HOME}/om-server/om/sbin/cleanSwitchScheduler.sh 1
+If information similar to the following is displayed, the switch is successful:
+Will change scheduler type to SUPERIOR +Start to delete all tenant resource. +End to delete all tenant resource. +Start to delete all resource pool. +End to delete all resource pool. +... +End to switch scheduler by reset.
+ - Run the following command to switch from the Capacity scheduler to the Superior scheduler:
sh ${BIGDATA_HOME}/om-server/om/sbin/cleanSwitchScheduler.sh 0
+If information similar to the following is displayed, the switch is successful:
+Will change scheduler type to CAPACITY +Start to delete all tenant resource. +End to delete all tenant resource. +Start to delete all resource pool. +End to delete all resource pool. +... +End to switch scheduler by reset.
+
+ You can query the scheduler switching logs on the active management node.
+- ${BIGDATA_LOG_HOME}/controller/aos/clean_switch_scheduler.log
- ${BIGDATA_LOG_HOME}/controller/aos/aos.log
- ${BIGDATA_LOG_HOME}/controller/aos/plugin.log
- Run the following command to switch from the Capacity scheduler to the Superior scheduler:
Switching from the Capacity Scheduler to the Superior Scheduler
Impact on the System
+- Because the ResourceManager is restarted during scheduler switching, submitting jobs to YARN will fail at that time.
- During scheduler switching, tasks in a job being executed on YARN will continue, but new tasks cannot be started.
- After scheduler switching is complete, jobs executed on YARN may fail, causing service interruptions.
- After scheduler switching is complete, parameters of the Superior scheduler are used for tenant management.
- After scheduler switching is complete, tenant queues whose capacity is 0 in the Capacity scheduler cannot be allocated resources in the Superior scheduler. As a result, jobs submitted to these tenant queues fail to be executed. Therefore, you are advised not to set the capacity of a tenant queue to 0 in the Capacity scheduler.
- After scheduler switching is complete, you cannot add or delete resource pools, YARN node labels, or tenants during the observation period. If such an operation is performed, the scheduler cannot be rolled back to the Capacity scheduler.+
The recommended observation period for scheduler switching is one week. If resource pools, YARN node labels, or tenants are added or deleted during this period, the observation period ends immediately.
+ - The scheduler rollback may cause the loss of partial or all YARN job information.
Procedure
+- Modify YARN service parameters and ensure that the YARN service status is normal.
- Log in to MRS Manager as an administrator.
- Log in to MRS Manager and choose Cluster > Services > Yarn. Click Configurations then All Configurations, search for yarn.resourcemanager.webapp.pagination.enable, and check whether the value is true.
- Choose Cluster > Name of the desired cluster > Services, and check whether the YARN service status is normal.
- Log in to the active management node as user omm.
- Switch the scheduler.
The following switching modes are available:
0: converts the Capacity scheduler configurations into the Superior scheduler configurations and then switches the Capacity scheduler to the Superior scheduler.
diff --git a/docs/mrs/umn/admin_guide_000157.html b/docs/mrs/umn/admin_guide_000157.html index 3b6d375c..e90a67e6 100644 --- a/docs/mrs/umn/admin_guide_000157.html +++ b/docs/mrs/umn/admin_guide_000157.html @@ -11,7 +11,7 @@ - When applying for certificates from the certificate administrator, you have provided the password for accessing the key file and applied for the certificate files in CRT, CER, CERT, and PEM formats and the key files in KEY and PEM formats. The requested certificates must have the issuing function.
Procedure
- Log in to MRS Manager and choose System > Certificate.
- Click
on the right of Upload Certificate. In the file selection window, browse to select the obtained TAR package of the certificate files. - Click Upload.
Manager uploads the compressed package and automatically imports the package.
- - After the certificate is imported, the system displays a message asking you to synchronize the cluster configuration and restart the web service for the new certificate to take effect. Click OK.
- In the displayed dialog box, enter the password of the current login user and click OK. The cluster configuration is automatically synchronized and the web service is restarted.
- After the cluster is restarted, enter the URL for accessing MRS Manager in the address box of the browser and check whether the MRS Manager web page can be successfully displayed.
- Log in to MRS Manager.
- Choose Cluster, click the name of the target cluster, choose Dashboard, click More, and select Restart.
- In the displayed dialog box, enter the password of the current login user and click OK.
- After the certificate is imported, the system displays a message asking you to synchronize the cluster configuration and restart the web service for the new certificate to take effect. Click OK.
- In the displayed dialog box, enter the password of the current login user and click OK. The cluster configuration is automatically synchronized and the web service is restarted.
- After the cluster is restarted, enter the URL for accessing MRS Manager in the address box of the browser and check whether the MRS Manager web page can be successfully displayed.
- Log in to MRS Manager.
- Choose Cluster, click the name of the target cluster, choose Dashboard, click More, and select Restart. (For MRS 3.3.0 or later, choose More > Restart in the upper right corner of the home page.)
- In the displayed dialog box, enter the password of the current login user and click OK.
diff --git a/docs/mrs/umn/admin_guide_000162.html b/docs/mrs/umn/admin_guide_000162.html index 87806540..25c1344e 100644 --- a/docs/mrs/umn/admin_guide_000162.html +++ b/docs/mrs/umn/admin_guide_000162.html @@ -48,6 +48,14 @@Port number of the Kpasswd service.
Reset LDAP Account Password
+Machine-machine users (cn=krbadmin,ou=Users,dc=hadoop,dc=com and cn=krbkdc,ou=Users,dc=hadoop,dc=com) used by Kerberos to access LDAP.
+If this parameter is selected, the passwords will be replaced by random passwords.
+NOTE:+This parameter is available only in MRS 3.1.2 or later.
+
Port number of the LDAP service.
Reset LDAP Account Password
+Machine-machine users (cn=root,dc=hadoop,dc=com and cn=pg_search_dn,ou=Users,dc=hadoop,dc=com) used by LDAP for data management, synchronization, and status check.
+If this parameter is selected, the passwords will be replaced by random passwords.
+NOTE:+This parameter is available only in MRS 3.1.2 or later.
+ - Permission Verification Policies
- - User Account ListUser Account ListSystem User Description
+ - User Account List
- Default Permission Information
diff --git a/docs/mrs/umn/admin_guide_000239.html b/docs/mrs/umn/admin_guide_000239.html
index e4c0c9fb..3e101b28 100644
--- a/docs/mrs/umn/admin_guide_000239.html
+++ b/docs/mrs/umn/admin_guide_000239.html
@@ -1,6 +1,6 @@
- - IoTDB administrator permissions:
- Creates or deletes a storage group.
- Uses TTL.
- IoTDB data operation permissions:
- Creates, modifies, and deletes a time sequence.
- Writes, reads, and deletes data in a time sequence.
- Views user or role permission information.
- Grants or revokes permissions to or from a user or role.NOTE:
In a common cluster, the IoTDB service retains the open-source feature. The default username is root, and the default password is root. This user is an administrator and has all permissions, which cannot be assigned, revoked, or deleted.
+- Views user or role permission information.
- Grants or revokes permissions to or from a user or role.
NOTE:In a common cluster, the IoTDB service retains the open-source feature. The default username is root, and the default password is root (for MRS 3.3.0 or later, the default password is Iotdb@123). This user is an administrator and has all permissions, which cannot be assigned, revoked, or deleted.
- HDFS
The dfs.permissions.superusergroup parameter specifies the administrator group with the highest permission on the HDFS. The default value is supergroup.
- Spark2x and Corresponding Multi-Instances
The spark.admin.acls parameter specifies the administrator list of the Spark2x. Members in the list are authorized to manage all Spark tasks. Users not added in the list cannot manage all Spark tasks. The default value is admin.
diff --git a/docs/mrs/umn/admin_guide_000246.html b/docs/mrs/umn/admin_guide_000246.html index 6982464c..af88b4f5 100644 --- a/docs/mrs/umn/admin_guide_000246.html +++ b/docs/mrs/umn/admin_guide_000246.html @@ -1,9 +1,9 @@Internal an Internal System User
-Scenario
If the service is abnormal, the internal user of the system may be locked. Unlock the user promptly, or the cluster cannot run properly. For the list of system internal users, see User Account ListUser Account ListSystem User Description in . The internal user of the system cannot be unlocked using MRS Manager.
+-Scenario
If the service is abnormal, the internal user of the system may be locked. Unlock the user promptly, or the cluster cannot run properly. For the list of system internal users, see User Account List in . The internal user of the system cannot be unlocked using MRS Manager.
Prerequisites
Obtain the default password of the LDAP administrator cn=root,dc=hadoop,dc=com by referring to User Account ListUser Account ListSystem User Description in .
+Prerequisites
Obtain the default password of the LDAP administrator cn=root,dc=hadoop,dc=com by referring to User Account List in .
Procedure
- Use the following method to confirm whether the internal system username is locked:
- OLdap port number obtaining method:
- Log in to MRS Manager, choose .
- The LDAP Listening Port parameter value is oldap port.
- Domain name obtaining method:
- Log in to MRS Manager, choose System > Permission > Domain and Mutual Trust.
- The Local Domain parameter value is the domain name.
For example, the domain name of the current system is 9427068F-6EFA-4833-B43E-60CB641E5B6C.COM.
diff --git a/docs/mrs/umn/admin_guide_000255.html b/docs/mrs/umn/admin_guide_000255.html index 5fc36a4b..17c54a89 100644 --- a/docs/mrs/umn/admin_guide_000255.html +++ b/docs/mrs/umn/admin_guide_000255.html @@ -1,7 +1,9 @@Changing the Passwords of the LDAP Administrator and the LDAP User (Including OMS LDAP)
-Scenario
It is recommended that the administrator periodically changes the passwords of LDAP administrator cn=root,dc=hadoop,dc=com and LDAP user cn=pg_search_dn,ou=Users,dc=hadoop,dc=com to improve the system O&M security.
++ This section applies only to MRS 3.1.0. For later versions, see Modifying OMS Service Configuration Parameters.
+Scenario
It is recommended that the administrator periodically changes the passwords of LDAP administrator cn=root,dc=hadoop,dc=com and LDAP user cn=pg_search_dn,ou=Users,dc=hadoop,dc=com to improve the system O&M security.
If the passwords are changed, the password of the OMS LDAP administrator or user is changed as well.
diff --git a/docs/mrs/umn/admin_guide_000256.html b/docs/mrs/umn/admin_guide_000256.html index 98d5446d..56eb380e 100644 --- a/docs/mrs/umn/admin_guide_000256.html +++ b/docs/mrs/umn/admin_guide_000256.html @@ -1,7 +1,9 @@ If the cluster is upgraded from an early version to a latest version, the LDAP administrator password will inherit the password policy of the old cluster. To ensure system security, you are advised to change the password after the cluster upgrade.
Changing the Password for the LDAP Administrator
-Scenario
It is recommended that the administrator periodically changes the passwords of LDAP administrator accounts cn=krbkdc,ou=Users,dc=hadoop,dc=com and cn=krbadmin,ou=Users,dc=hadoop,dc=com to improve the system O&M security.
++ This section applies only to MRS 3.1.0. For later versions, see Modifying OMS Service Configuration Parameters.
+Scenario
It is recommended that the administrator periodically changes the passwords of LDAP administrator accounts cn=krbkdc,ou=Users,dc=hadoop,dc=com and cn=krbadmin,ou=Users,dc=hadoop,dc=com to improve the system O&M security.
Impact on the System
- You need to restart the KrbServer service after changing the password.
- After the password is changed, check whether the LDAP administrator accounts cn=krbkdc,ou=Users,dc=hadoop,dc=com and cn=krbadmin,ou=Users,dc=hadoop,dc=com are locked, run the following command on the active management node of the cluster to check whether krbkdc is locked (the method for user krbadmin is similar):
OLdap port number obtaining method:
- Log in to MRS Manager, choose :
- The LDAP Listening Port parameter value is oldap port.
- Updating SSH Keys for User omm
+
- Changing the Timeout Duration of the Manager Page
+diff --git a/docs/mrs/umn/admin_guide_000407.html b/docs/mrs/umn/admin_guide_000407.html new file mode 100644 index 00000000..d0918e0d --- /dev/null +++ b/docs/mrs/umn/admin_guide_000407.html @@ -0,0 +1,29 @@ + + + +SQL Inspector
+ +++ +++ diff --git a/docs/mrs/umn/admin_guide_000408.html b/docs/mrs/umn/admin_guide_000408.html new file mode 100644 index 00000000..7a5cf3af --- /dev/null +++ b/docs/mrs/umn/admin_guide_000408.html @@ -0,0 +1,22 @@ + + +-
+
- Overview
+
+ - Adding an SQL Inspection
+
+ - Configuring Hive SQL Inspection
+
+ - Configuring ClickHouse SQL Inspection
+
+ - Configuring HetuEngine SQL Inspection
+
+ - Configuring Spark SQL Inspection
+
+
++Parent topic: MRS Manager Operation Guide (Applicable to 3.x)+Overview
++SQL engines in the big data field are emerging one after another. In addition to a wide range of solutions, some problems are exposed. For example, the quality of SQL input statements is uneven, SQL problems are difficult to locate, and large SQL statements consume too many resources.
+Low-quality SQL statements pose unexpected impacts on the data analysis platform, degrading system performance or platform stability.
++ This function is supported only by MRS 3.3.0 or later.
++Function Description
MRS allows you to configure inspection rules for mainstream SQL engines (Hive, Spark, HetuEngine, and ClickHouse). MRS can identify typical large SQL queries and low-quality SQL statements and intercepts them before execution or block them during execution. Users do not need to change how they submit SQL statements or change SQL syntax. Service modifications are not required and inspection is easy to implement.
+- You can configure SQL inspection rules on the UI that also allows you to query and modify the rules.
- During query response and execution, each SQL engine proactively inspects SQL statements based on the rules.
- Administrators can select to display hints on, intercept, or block SQL statements. The system logs SQL inspection events in real time for SQL audit. O&M engineers can analyze the logs, evaluate SQL statement quality on the live network, detect target statements, and take effective measures.
SQL inspection rules are classified into the following types:
+- Static interception: The system displays hints on or intercepts SQL statements based on SQL syntax rules.
- Dynamic interception: The system displays hints on or intercepts SQL statements based on rules of data table statistics and metadata information.
- Runtime Blocking: The system blocks SQL statements based on system states (such as CPU, memory, and I/O) during the runtime of the SQL statements.
SQL requests that meet the static and dynamic interception rules can be intercepted, and the system gives hints for processing the statements properly. If a SQL request meets the blocking rule, the system blocks the SQL task.
++Rules and Restrictions
- A SQL inspection rule can be associated with multiple SQL engines, and different threshold parameters can be configured for each service.
- A SQL inspection rule can be associated with multiple tenants. A rule takes effect only for associated tenants.
++ diff --git a/docs/mrs/umn/admin_guide_000409.html b/docs/mrs/umn/admin_guide_000409.html new file mode 100644 index 00000000..64c25b45 --- /dev/null +++ b/docs/mrs/umn/admin_guide_000409.html @@ -0,0 +1,330 @@ + + +++Parent topic: SQL Inspector+Adding an SQL Inspection
+++Scenario
You can add rules for specified tenants and SQL engines on FusionInsight Manager. The system will display hints on, intercept, or block SQL requests matched by the rules.
++ Exercise caution when you add or modify a SQL inspection rule for a cluster, enable a rule, and set the threshold. An improper rule may cause upper-layer service interruption.
++Adding a Rule
- Log in to FusionInsight Manager as a user with the Manager administrator rights.
- Click Cluster and choose SQL Inspector. The SQL Inspector page is displayed.
You can click View Supported Rules to view all SQL inspection rules supported by the current cluster.
+ - Click Add Rule. After the password of the current user is verified, the Add Rule page is displayed.
- Set the required parameters and click OK.
+
+
+Parameter
+Description
+Name
+Name of a SQL inspection rule
+ID
+Rule ID
+For details about meaning of the rules corresponding to the IDs, see Table 1.
+Tenant
+Click Add to select the name of the tenant to which the current rule will be associated.
+If you need to add a new tenant, plan and create a cluster tenant by referring to Tenant Resources.
+Services and Actions
+Click Add to specify the SQL engine to which this rule will be associated with and set the threshold parameters of the rule.
+Each rule can be associated with one SQL engine. If you want to configure a rule for other SQL engines, add new rules.
+- Service: Select the SQL engine associated with the current rule.
- If an SQL request meets the rule, the system performs the following operations:
- Hint: Record logs and display a hint for handling the SQL request. If the rule has parameters, you need to configure the threshold.
- Intercept: Intercept the SQL request that meets the rule. If the rule has parameters, you need to configure the threshold.
- Block: Block the SQL request that meets the rule. If the rule has parameters, you need to configure the threshold.NOTE:+
For static and dynamic interception rules, Hint and Block operations are supported. For blocking rules, only the Block operation is supported.
+
- View the added prevention rule on the SQL Defense page. The rule takes effect dynamically.
To adjust the current rule, click Modify in the Operation column of the row that contains the target rule. After the user password is verified, you can modify rule parameters.
+Figure 1 Viewing SQL inspection rules+
+MRS SQL Inspection Rules
++
+Table 1 MRS SQL inspection rules ID
+Description
+Engine
+Threshold
+Example SQL Statement
+static_0001
+Check whether the number of occurrences of count(distinct) in the SQL statement exceeds the limit.
+- Hive
- Spark
- HetuEngine
Number of occurrences of count(distinct)
+Recommended value: 10
+SELECT COUNT(DISTINCT deviceId), COUNT(DISTINCT collDeviceId)
+FROM table
+GROUP BY deviceName, collDeviceName, collCurrentVersion;
+static_0002
+Check whether the not in <subquery> statement is used in the SQL statement.
+- Hive
- Spark
- HetuEngine
N/A
+SELECT *
+FROM Orders o
+WHERE Orders.Order_ID not in (Select Order_ID
+FROM HeldOrders h
+where h.order_id = o.order_id);
+static_0003
+Check whether the number of joins in the SQL statement exceeds the limit.
+- Hive
- Spark
- HetuEngine
Number of joins
+Recommended value: 20
+N/A
+static_0004
+Check whether the number of the union all operators in a SQL statement exceeds the threshold.
+- Hive
- Spark
- HetuEngine
Number of the union all operators in a statement
+Recommended value: 20
+select * from tables t1
+union all select * from tables t2
+union all select * from tables t3
+union all select * from tables t4
+union all select * from tables t5
+union all select * from tables t6
+union all select * from tables t7
+union all select * from tables t8
+union all select * from tables t9;
+static_0005
+The number of subquery nesting layers exceeds the limit.
+- Hive
- Spark
- HetuEngine
Maximum number of nested subqueries
+Recommended value: 20
+select * from (
+with temp1 as (select * from tables)
+select * from temp1);
+static_0006
+Check whether the length of the SQL statement string exceeds the upper limit.
+- Hive
- Spark
- HetuEngine
Length of the SQL string, in KB
+Recommended value: 10
+N/A
+static_0007
+Check whether the Cartesian product exists when multiple tables are associated.
+- Hive
- Spark
- HetuEngine
N/A
+select * from A,B;
+static_0008
+Check whether alter table update operation is performed at the cluster level (on cluster).
+ClickHouse
+N/A
+alter table testtb1 on cluster default_cluster update price=10.0 where id='100'
+static_0009
+Check whether alter table delete operation is performed at the cluster level (on cluster).
+ClickHouse
+N/A
+alter table testtb1 on cluster default_cluster delete where id ='10'
+static_0010
+Check whether the alter table add column operation is performed at the cluster level (on cluster).
+ClickHouse
+N/A
+alter table testtb1 on cluster default_cluster add column testc String
+static_0011
+Check whether the alter table drop column operation is performed at the cluster level (on cluster).
+ClickHouse
+N/A
+alter table testtb1 on cluster default_cluster drop column testc
+static_0012
+Check whether the optimize final operation is performed at the cluster level (on cluster).
+ClickHouse
+N/A
+optimize table testtb1 on cluster default_cluster final
+static_0013
+Check whether the drop table operation is performed at the cluster level (on cluster).
+ClickHouse
+N/A
+drop table testtb1 on cluster default_cluster;
+static_0014
+Check whether the truncate table operation is performed at the cluster level (on cluster).
+ClickHouse
+N/A
+truncate table testtb1 on cluster default_cluster;
+dynamic_0001
+Check whether the number of scanned files exceeds the limit.
+- Hive
- Spark
- HetuEngine
Number of files that will be scanned or have been scanned
+Recommended value: 100,000
+SELECT ss_ticket_number FROM store_sales WHERE ss_ticket_number=72291252 LIMIT 10;
+dynamic_0002
+Check whether the number of partitions involved in operations (select, delete, update, and alter) on a table exceeds the upper limit.
+- Hive
- Spark
- HetuEngine
- ClickHouse
Number of partitions involved in the delete or alter operation
+Recommended value: 10,000
+DELETE FROM table_name WHERE column_name = value
+dynamic_0003
+When the right table of a join is a distributed table, check whether the data volume of the right table exceeds the upper limit.
+ClickHouse
+Number of rows in the right table when the join operation is performed.
+Recommended value: 100,000,000
+SELECT name, text FROM table_1 JOIN table_2 ON table_1.Id = table_2.Id
+running_0001
+Check whether the number of result rows returned by the Select statement to the client exceeds the upper limit.
+- Hive
- Spark
- HetuEngine
- ClickHouse
Number of rows in the query result
+Recommended value: 100,000
+select * from table
+running_0002
+Check whether the peak memory usage of the SQL statement exceeds the absolute value limit.
+- Hive
- Spark
- HetuEngine
- ClickHouse
Memory occupied by SQL running, in MB
+N/A
+running_0003
+Check whether the running duration of the SQL statement exceeds the upper limit.
+- Hive
- Spark
- HetuEngine
- ClickHouse
SQL running duration threshold, in seconds
+N/A
+running_0004
+The amount of data scanned by the SQL statements.
+- Hive
- Spark
- HetuEngine
- ClickHouse
Amount of data scanned by the SQL statement, in GB
+Recommended value: 10,240
+N/A
+++ diff --git a/docs/mrs/umn/admin_guide_000410.html b/docs/mrs/umn/admin_guide_000410.html new file mode 100644 index 00000000..59784d46 --- /dev/null +++ b/docs/mrs/umn/admin_guide_000410.html @@ -0,0 +1,17 @@ + + +++Parent topic: SQL Inspector+Changing the Timeout Duration of the Manager Page
++FusionInsight Manager allows you to configure the timeout duration of the Manager page based on service requirements. You must properly set the timeout duration to prevent information leakage in long-time exposure of the web page.
++ This function is supported only by MRS 3.3.0 or later.
++Changing the Timeout Duration of the Manager Page
- Log in to FusionInsight Manager.
- Choose System > OMS.
- In the list, locate the row that contains tomcat and click Modify Configuration.
- On the displayed page, set Session Timeout as required and click OK.+
- Set the minimum session duration based on service requirements. Otherwise, there will be security risks.
- Currently, you cannot use the method described as follows to change the timeout duration of component web UIs.
++ diff --git a/docs/mrs/umn/admin_guide_000412.html b/docs/mrs/umn/admin_guide_000412.html new file mode 100644 index 00000000..9c504469 --- /dev/null +++ b/docs/mrs/umn/admin_guide_000412.html @@ -0,0 +1,42 @@ + + +++Parent topic: Security Hardening+Configuring Hive SQL Inspection
+++Scenario
You can configure rules for Hive SQL inspection on FusionInsight Manager and configure rule parameters as you need.
++Prerequisites
- The cluster client that contains the Hive service has been installed in the /opt/hadoopclient directory.
- The Hive service of the cluster is running properly.
- For a cluster with Kerberos authentication enabled, a user with Hive operation permissions has been created.
+Constraints
- By default, SQL inspection rules need 5 seconds to take effect dynamically. After the queue is modified, it takes 10 minutes for Hive inspection rules to be reloaded.
- Interception and blocking rules will interrupt SQL tasks, so you need to set parameters of these rules properly based on the site requirements.
- For the rule dynamic_0001 (the number of files scanned by SQL statements exceeds the threshold), when the Spark and Tez engines reach the threshold, interception logs are printed in Yarn task logs and cannot be output on the Beeline client.
- Blocking rules have execution latency. For example, if the running_0004 rule is used and the threshold of the scanned data volume is 10 GB, the statement may be blocked when the data volume is 15 GB or higher due to the determination period and task concurrency.
+Procedure
- Log in to FusionInsight Manager, click Cluster, and choose SQL Inspector. The SQL Inspector page is displayed.
- Add rules for Hive by referring to Adding an SQL Inspection.
For details about the rules supported by the Hive SQL engine, see MRS SQL Inspection Rules.
+For example, add a rule whose ID is static_0001 to check whether count distinct appears more than two times in the SQL statement. If so, the system displays a hint.
+Figure 1 Adding a Hive SQL inspection rule+
- Log in to the node where the Hive client is installed and run the following command to switch to the client installation directory.
cd /opt/hadoopclient
+Run the following command to set environment variables:
+source bigdata_env
+Run the following command to authenticate the current user. Skip this step if Kerberos authentication is disabled for the cluster (the cluster is in normal mode).
+kinit Component service user who has the Hive operation permission
+ - Run the following command to log in to the Hive client:
beeline
+ - Run the following commands to create a table and import data to the table.
drop table if exists hivetb;
+create table hivetb(a int,b int);
+insert into hivetb select 1,11;
+insert into hivetb select 2,22;
+ - Run the following SQL statement to check whether the current rule takes effect:
select count(distinct a),count(distinct b) from hivetb;
+If the number of times count distinct appears in the statement exceeds the threshold configured in 2, the following information is displayed:
+... +WARN : STATIC_0001 The count(distinct X) times exceeds the limit : 2, current count distinct times : 2 +...
+If the operation set in the rule is Block, the statement fails to be executed and the following information is displayed:+... +Error: Error while compiling statement: FAILED: RuleException STATIC_0001 The count(distinct X) times exceeds the limit : 2, current count distinct times : 2 (state=42000,code=40000) +...
++ - For more Hive SQL inspection rules, see MRS SQL Inspection Rules.
- You can also obtain the SQL inspection rules via logs which are stored in /var/log/Bigdata/audit/hive/hiveserver/queryinfo.log.
++ diff --git a/docs/mrs/umn/admin_guide_000413.html b/docs/mrs/umn/admin_guide_000413.html new file mode 100644 index 00000000..6452c975 --- /dev/null +++ b/docs/mrs/umn/admin_guide_000413.html @@ -0,0 +1,46 @@ + + +++Parent topic: SQL Inspector+Configuring ClickHouse SQL Inspection
+++Scenario
You can configure rules for ClickHouse SQL inspection on FusionInsight Manager and configure rule parameters as you need.
++Prerequisites
- The cluster client that contains the ClickHouse service has been installed in the /opt/hadoopclient directory.
- The ClickHouse logical cluster is running properly.
- For clusters with Kerberos authentication enabled, you need to create a service user who has the permission to operate the ClickHouse table. For example, create a human-machine user clickhouseuser.
- A tenant associated with the ClickHouse service has been created and associated with the ClickHouse service user. For details, see Creating Tenants
+Constraints
- The default dynamic validity period of a rule is 1 minute.
- Interception and blocking rules will interrupt SQL queries, so you need to set parameters of these rules properly based on the site requirements.
- After configuring ClickHouse rules, you need to log in to the client again for the rules to take effect.
+Procedure
- Log in to FusionInsight Manager, click Cluster, and choose SQL Inspector. The SQL Inspector page is displayed.
- Add rules for ClickHouse by referring to Adding an SQL Inspection.
For details about the rules supported by the ClickHouse SQL engine, see MRS SQL Inspection Rules.
+For example, add a rule whose ID is static_0008 and checks whether a SQL statement executes the cluster-level table update operation. If so, the system displays a hint.
+Figure 1 Adding a ClickHouse SQL inspection rule+
- Log in to the node where the ClickHouse client is installed and run the following command to switch to the client installation directory.
cd /opt/hadoopclient
+Run the following command to set environment variables:
+source bigdata_env
+ - If the current cluster is in security mode (Kerberos authentication is enabled), run the following command to authenticate the current user. The current user must have the permission to create ClickHouse tables. If the current cluster is in normal mode (Kerberos authentication is disabled), skip this step.
kinit Component service user
+Example: kinit clickhouseuser
+ - Use the ClickHouse client to connect to the ClickHouse server.
Security mode
+clickhouse client --host IP address of the ClickHouseServer instance --port 9440 --secure
+Normal clusters:
+clickhouse client --host IP address of the ClickHouseServer instance--user Username --password --port 9000
+Enter the password.
+ - Run the following statements to create a data table:
CREATE DATABASE cktest ON CLUSTER default_cluster;
+CREATE TABLE cktest.test2 ON CLUSTER default_cluster ( `EventDate` DateTime, `CounterID` UInt32, `UserID` UInt32, `ver` UInt16 ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/cktest/test2', '{replica}') PARTITION BY toYYYYMM(EventDate) ORDER BY (EventDate, intHash32(UserID));
+CREATE TABLE cktest.test2_dir ON CLUSTER default_cluster as cktest.test2 ENGINE = Distributed(default_cluster, cktest, test2, rand());
+ - Run the following command to insert data to the table:
insert into cktest.test2 values('2023-08-01',111,111,111);
+insert into cktest.test2 values('2023-08-02',222,111,111);
+ - Run the following SQL statement for the created table to check whether the rule takes effect:
alter table cktest.test2 on cluster default_cluster update CounterID = toUInt32(222) where EventDate='2023-08-01' ;
+... +<Warning> SQLDefender: Distributed DDL ALTER UPDATE queries are undesirable. +...
+If the operation set in the rule is Intercept, the statement fails to be executed and the following information is displayed:
+... +DB::Exception: Distributed DDL ALTER TABLE UPDATE queries are undesirable..(QUERY_IS_PROHIBITED) +...
+ +
++ diff --git a/docs/mrs/umn/admin_guide_000414.html b/docs/mrs/umn/admin_guide_000414.html new file mode 100644 index 00000000..05fbf475 --- /dev/null +++ b/docs/mrs/umn/admin_guide_000414.html @@ -0,0 +1,48 @@ + + +++Parent topic: SQL Inspector+Configuring Spark SQL Inspection
+++Scenario
You can configure rules for Spark SQL inspection on FusionInsight Manager and configure rule parameters as you need.
++Prerequisites
- The cluster client that contains the Spark service has been installed in the /opt/hadoopclient directory.
- Spark is running properly.
- A tenant, for example, sparkstatic1 has been added to Tenant Resources. For details, see Creating Tenants.
- For a cluster with Kerberos authentication enabled, a service user has been created, for example, user sparkuser. The user belongs to groups hive, hadoop, and supergroup, the primary group is hive, and the bound role is set to sparkstatic1.
+Constraints
- The default dynamic validity period of a rule is 6 minutes.
- Only SQL jobs are supported.
- Interception and blocking rules will interrupt SQL queries, so you need to set parameters of these rules properly based on the site requirements.
- The static rule static_0007 is not required for Spark because it has a Cartesian product restriction (controlled by spark.sql.crossJoin.enabled, which is true by default). If the spark parameter is false, the static_0007 rule will not take effect.
- Dynamic rules do work on carbon tables.
- The dynamic rule dynamic_0002 supports SELECT, ALTER TABLE ADD PARTITION and ALTER TABLE DROP PARTITION. If you use a batch deletion statement that contains judgment conditions, for example, ALTER TABLE DROP PARTITION (pt < 10), the statement will be intercepted before the dynamic_0002 rule because the number of partitions is limited by spark.sql.dropPartitionsInBatch.limit, which is defaulted to 1000.
- Blocking rules has execution latency. For example, if the running_0004 rule is used and the threshold of the scanned data volume is 10 GB, the statement may be blocked when the data volume is 15 GB or higher due to the determination period and task concurrency.
- If a job does not exceed the threshold until the last several tasks are complete before the block action is triggered, the job cannot be canceled.
- Blocking rule running_0004: The SQL execution duration includes the execution duration on the Driver side and the job running duration. When the SQL execution is blocked on the Driver side, the job cannot be canceled even if the execution duration exceeds the value of interruption threshold. This problem may occur when INSERT OVERWRITE is performed on a large number of partitions where storage and compute are decoupled.
+ +Procedure
- Log in to FusionInsight Manager, click Cluster, and choose SQL Inspector. The SQL Inspector page is displayed.
- Add rules for Spark by referring to Adding an SQL Inspection.
For details about the rules supported by the Spark SQL engine, see MRS SQL Inspection Rules.
+For example, add a rule whose ID is static_0001 to check whether count distinct appears more than two times in the SQL statement. If so, the system displays a hint.
+Figure 1 Adding a Spark SQL inspection rule+
- Log in to the node where the Spark client is installed and run the following command to switch to the client installation directory.
cd /opt/hadoopclient
+Run the following command to set environment variables:
+source bigdata_env
+source Spark/component_env
+ - Perform user authentication for clusters in security mode (Kerberos authentication enabled). Skip this step for clusters in normal mode (Kerberos authentication disabled).
kinit Spark operation user
+Example:
+kinit sparkuser
+Enter the password as prompted and change the password upon your first login.
+ - Run the following command to log in to the spark-sql client:
cd opt/client/Spark/spark/bin
+./spark-sql
+ - Run the following SQL statement on the client to check whether the current rule takes effect:
Run the following statement to create a table:
+create table table1(id int, name string) stored as parquet
+Run the following statement to query data:
+select count(distinct id),count(distinct id),count(distinct id),count(distinct id),count(distinct id),count(distinct id) from table1;
+If the number of times count distinct appears in the statement exceeds the threshold configured in 2, the following information is displayed:
+WARNING: static_0001 Occurrence num of 'COUNT(DISTINCT)'(6) reaches the hint threshold(2)
+If the action set in the rule is Intercept, the following information is displayed:+Error in query: static_0001 Occurrence num of 'COUNT(DISTINCT)'(6) reaches the intercept threshold(2)
+In Spark Beeline, you can obtain SQL inspection details from logs.
+- Log in to FusionInsight Manager and choose Cluster > Services > Yarn. On the Dashboard page, click the link next to ResourceManager WebUI to enter the Yarn web UI.
- Click the ID of the target application on the All Applications page. The application details page is displayed.
+
- Click Logs of the application. On the displayed page, click stdout logs to view SQL inspection details.
+
+
+
+ - For more Spark SQL inspection rules, see MRS SQL Inspection Rules.
- You can view query info in the /opt/hadoopclient/Spark/spark/audit/query.log path if you are using the Spark client. The log contains detailed running information and the corresponding SQL inspection information.
++ diff --git a/docs/mrs/umn/admin_guide_000416.html b/docs/mrs/umn/admin_guide_000416.html new file mode 100644 index 00000000..3de830e1 --- /dev/null +++ b/docs/mrs/umn/admin_guide_000416.html @@ -0,0 +1,43 @@ + + +++Parent topic: SQL Inspector+Configuring HetuEngine SQL Inspection
+++Scenario
You can configure rules for HetuEngine SQL inspection on FusionInsight Manager and configure rule parameters as you need.
++Prerequisites
- The cluster client that contains the HetuEngine service has been installed in the /opt/hadoopclient directory.
- The HetuEngine service and compute instances are running properly.
- If Kerberos authentication has been enabled for the cluster, you need to create a HetuEngine user and grant related permissions to the user. In addition, you need to use Ranger to assign the user the permission to manage databases, tables, and columns of the data source.
+Constraints
- The default dynamic validity period of a rule is 5 minutes.
- Interception and blocking rules will interrupt SQL queries, so you need to set parameters of these rules properly based on the site requirements.
- Blocking rules are controlled by session-level parameters of the system. To configure blocking rules, service users must have the set session permission.
- For static rule static_0003, the total number of joins in queries does not include Semi joins and Anti joins.
- When prompt rules are configured for dynamic_0001 and dynamic_0002, prompt messages are recorded only in logs and are not displayed on the client.
- The client and server send asynchronous requests. For blocking rule Running_0001, after the server blocks the requests, the message "Query is gone " may be displayed on the client. In this case, you can view logs to check whether the requests are blocked.
+Procedure
- Log in to FusionInsight Manager, click Cluster, and choose SQL Inspector. The SQL Inspector page is displayed.
- Add rules for HetuEngine by referring to Adding an SQL Inspection.
For details about the rules supported by the HetuEngine SQL engine, see MRS SQL Inspection Rules.
+For example, add a rule whose ID is static_0001 to check whether count distinct appears more than two times in the SQL statement. If so, the system displays a hint.
+Figure 1 Adding a HetuEngine SQL inspection rule+
- Log in to the node where the HetuEngine client is installed and run the following command to switch to the client installation directory:
cd /opt/hadoopclient
+Run the following command to set environment variables:
+source bigdata_env
+ - Log in to the HetuEngine client based on the cluster authentication mode.
- In security mode, run the following command to authenticate the user and log in to the HetuEngine client:
+
hetu-cli --catalog hive --tenant default --schema default
+ - In normal mode, run the following command to log in to the HetuEngine client:
hetu-cli --catalog hive --tenant default --schema default --user hetu_test
++ hetu_test is a service user who has at least the tenant role specified by --tenant and cannot be an OS user.
+
- In security mode, run the following command to authenticate the user and log in to the HetuEngine client:
+
- Check whether the current rule takes effect.
Run the following statement to create a table:
+CREATE TABLE table1(id int, name varchar,rank int);
+INSERT INTO table1 VALUES(10,'sachin',1),(45,'rohit',2),(46,'rohit',3),(18,'virat',4),(25,'dhawan',5);
+Run the following statement to query data:
+select count(distinct id),count(distinct id),count(distinct id),count(distinct id),count(distinct id),count(distinct id) from table1;
+If the number of times count distinct appears in the statement exceeds the threshold configured in 2, the following information is displayed:
+WARNING: Occurrence number of 'COUNT(DISTINCT XX)' (6) reaches the hint limitation (2)
++ - If the action set in the rule is Intercept or Block, the following information may be displayed:
Intercepted. Reason: Occurrence number of 'COUNT(DISTINCT XX)' (6) reaches the interception limitation (2)
+ - You can query HetuEngine SQL inspection details in logs stored in hdfs://hacluster/hetuserverhistory/tenant/coordinator/application_ID/container_ID/yyyyMMdd/server.log.
- If warning information is required for JDBC secondary development, add the following configuration for the JDBC application:
statement = connection.prepareStatement(sql.trim()); +resultSet = statement.executeQuery(); +SQLWarning sqlWarning = statement.getWarnings();
+
- If the action set in the rule is Intercept or Block, the following information may be displayed:
++ diff --git a/docs/mrs/umn/en-us_image_0000001532448466.png b/docs/mrs/umn/en-us_image_0000001532448466.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001532448466.png differ diff --git a/docs/mrs/umn/en-us_image_0000001582927845.png b/docs/mrs/umn/en-us_image_0000001582927845.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001582927845.png differ diff --git a/docs/mrs/umn/en-us_image_0000001583087613.png b/docs/mrs/umn/en-us_image_0000001583087613.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001583087613.png differ diff --git a/docs/mrs/umn/en-us_image_0000001583127589.png b/docs/mrs/umn/en-us_image_0000001583127589.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001583127589.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971003602.png b/docs/mrs/umn/en-us_image_0000001971003602.png new file mode 100644 index 00000000..e85138f1 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971003602.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971010166.png b/docs/mrs/umn/en-us_image_0000001971010166.png new file mode 100644 index 00000000..8ef8b5d5 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971010166.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971010174.png b/docs/mrs/umn/en-us_image_0000001971010174.png new file mode 100644 index 00000000..0290a929 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971010174.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971010178.png b/docs/mrs/umn/en-us_image_0000001971010178.png new file mode 100644 index 00000000..da39cbc2 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971010178.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971074706.png b/docs/mrs/umn/en-us_image_0000001971074706.png new file mode 100644 index 00000000..bc15e827 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971074706.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971077930.png b/docs/mrs/umn/en-us_image_0000001971077930.png new file mode 100644 index 00000000..ae5e7587 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971077930.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971077934.png b/docs/mrs/umn/en-us_image_0000001971077934.png new file mode 100644 index 00000000..1455ca2e Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971077934.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971077942.png b/docs/mrs/umn/en-us_image_0000001971077942.png new file mode 100644 index 00000000..7029c8b4 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971077942.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971163390.png b/docs/mrs/umn/en-us_image_0000001971163390.png new file mode 100644 index 00000000..e56c7809 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971163390.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971169950.png b/docs/mrs/umn/en-us_image_0000001971169950.png new file mode 100644 index 00000000..e760e6a2 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971169950.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971169958.png b/docs/mrs/umn/en-us_image_0000001971169958.png new file mode 100644 index 00000000..8ef8b5d5 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971169958.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971169962.png b/docs/mrs/umn/en-us_image_0000001971169962.png new file mode 100644 index 00000000..07df372d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971169962.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971172618.png b/docs/mrs/umn/en-us_image_0000001971172618.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971172618.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971237698.png b/docs/mrs/umn/en-us_image_0000001971237698.png new file mode 100644 index 00000000..c8083eb7 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971237698.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971237702.png b/docs/mrs/umn/en-us_image_0000001971237702.png new file mode 100644 index 00000000..068ab5eb Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971237702.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971622178.png b/docs/mrs/umn/en-us_image_0000001971622178.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971622178.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971622210.png b/docs/mrs/umn/en-us_image_0000001971622210.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971622210.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971622238.png b/docs/mrs/umn/en-us_image_0000001971622238.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971622238.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648670.png b/docs/mrs/umn/en-us_image_0000001971648670.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648670.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648702.png b/docs/mrs/umn/en-us_image_0000001971648702.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648702.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648738.png b/docs/mrs/umn/en-us_image_0000001971648738.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648738.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648770.png b/docs/mrs/umn/en-us_image_0000001971648770.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648770.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648806.png b/docs/mrs/umn/en-us_image_0000001971648806.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648806.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648822.png b/docs/mrs/umn/en-us_image_0000001971648822.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648822.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648826.png b/docs/mrs/umn/en-us_image_0000001971648826.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648826.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648838.png b/docs/mrs/umn/en-us_image_0000001971648838.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648838.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648850.png b/docs/mrs/umn/en-us_image_0000001971648850.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648850.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971648854.png b/docs/mrs/umn/en-us_image_0000001971648854.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971648854.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971659200.png b/docs/mrs/umn/en-us_image_0000001971659200.png new file mode 100644 index 00000000..d6da7802 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971659200.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971659208.png b/docs/mrs/umn/en-us_image_0000001971659208.png new file mode 100644 index 00000000..b99e9cf8 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971659208.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971659216.png b/docs/mrs/umn/en-us_image_0000001971659216.png new file mode 100644 index 00000000..eda3d116 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971659216.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971781990.png b/docs/mrs/umn/en-us_image_0000001971781990.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971781990.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808406.png b/docs/mrs/umn/en-us_image_0000001971808406.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808406.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808442.png b/docs/mrs/umn/en-us_image_0000001971808442.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808442.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808474.png b/docs/mrs/umn/en-us_image_0000001971808474.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808474.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808510.png b/docs/mrs/umn/en-us_image_0000001971808510.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808510.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808542.png b/docs/mrs/umn/en-us_image_0000001971808542.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808542.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808574.png b/docs/mrs/umn/en-us_image_0000001971808574.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808574.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808582.png b/docs/mrs/umn/en-us_image_0000001971808582.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808582.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808590.png b/docs/mrs/umn/en-us_image_0000001971808590.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808590.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808602.png b/docs/mrs/umn/en-us_image_0000001971808602.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808602.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808618.png b/docs/mrs/umn/en-us_image_0000001971808618.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808618.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971808622.png b/docs/mrs/umn/en-us_image_0000001971808622.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971808622.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971818972.png b/docs/mrs/umn/en-us_image_0000001971818972.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971818972.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971818980.png b/docs/mrs/umn/en-us_image_0000001971818980.png new file mode 100644 index 00000000..56e87891 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971818980.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971818984.png b/docs/mrs/umn/en-us_image_0000001971818984.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971818984.png differ diff --git a/docs/mrs/umn/en-us_image_0000001971955678.png b/docs/mrs/umn/en-us_image_0000001971955678.png new file mode 100644 index 00000000..e9d92446 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001971955678.png differ diff --git a/docs/mrs/umn/en-us_image_0000001972894010.png b/docs/mrs/umn/en-us_image_0000001972894010.png new file mode 100644 index 00000000..487cc0e7 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001972894010.png differ diff --git a/docs/mrs/umn/en-us_image_0000001972894046.png b/docs/mrs/umn/en-us_image_0000001972894046.png new file mode 100644 index 00000000..72cf81ad Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001972894046.png differ diff --git a/docs/mrs/umn/en-us_image_0000001972894054.png b/docs/mrs/umn/en-us_image_0000001972894054.png new file mode 100644 index 00000000..2be7d4c7 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001972894054.png differ diff --git a/docs/mrs/umn/en-us_image_0000001972894058.png b/docs/mrs/umn/en-us_image_0000001972894058.png new file mode 100644 index 00000000..cb06c43a Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001972894058.png differ diff --git a/docs/mrs/umn/en-us_image_0000001972894062.png b/docs/mrs/umn/en-us_image_0000001972894062.png new file mode 100644 index 00000000..3612fb59 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001972894062.png differ diff --git a/docs/mrs/umn/en-us_image_0000001972894066.png b/docs/mrs/umn/en-us_image_0000001972894066.png new file mode 100644 index 00000000..f56b7f52 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001972894066.png differ diff --git a/docs/mrs/umn/en-us_image_0000001973053750.png b/docs/mrs/umn/en-us_image_0000001973053750.png new file mode 100644 index 00000000..f889fa43 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001973053750.png differ diff --git a/docs/mrs/umn/en-us_image_0000001973053758.png b/docs/mrs/umn/en-us_image_0000001973053758.png new file mode 100644 index 00000000..a7cdffa4 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001973053758.png differ diff --git a/docs/mrs/umn/en-us_image_0000001973053802.png b/docs/mrs/umn/en-us_image_0000001973053802.png new file mode 100644 index 00000000..4fa25053 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001973053802.png differ diff --git a/docs/mrs/umn/en-us_image_0000001973053806.png b/docs/mrs/umn/en-us_image_0000001973053806.png new file mode 100644 index 00000000..33546497 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001973053806.png differ diff --git a/docs/mrs/umn/en-us_image_0000001973053810.png b/docs/mrs/umn/en-us_image_0000001973053810.png new file mode 100644 index 00000000..90d36ab7 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001973053810.png differ diff --git a/docs/mrs/umn/en-us_image_0000001973053822.png b/docs/mrs/umn/en-us_image_0000001973053822.png new file mode 100644 index 00000000..617a0f4f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000001973053822.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007523933.png b/docs/mrs/umn/en-us_image_0000002007523933.png new file mode 100644 index 00000000..956d75d2 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007523933.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007530501.png b/docs/mrs/umn/en-us_image_0000002007530501.png new file mode 100644 index 00000000..d352ad98 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007530501.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007530505.png b/docs/mrs/umn/en-us_image_0000002007530505.png new file mode 100644 index 00000000..877fbcfb Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007530505.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007530509.png b/docs/mrs/umn/en-us_image_0000002007530509.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007530509.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007533201.png b/docs/mrs/umn/en-us_image_0000002007533201.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007533201.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007643377.png b/docs/mrs/umn/en-us_image_0000002007643377.png new file mode 100644 index 00000000..7b511e47 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007643377.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007649989.png b/docs/mrs/umn/en-us_image_0000002007649989.png new file mode 100644 index 00000000..d75fdf96 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007649989.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007649997.png b/docs/mrs/umn/en-us_image_0000002007649997.png new file mode 100644 index 00000000..d0aa219a Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007649997.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007650001.png b/docs/mrs/umn/en-us_image_0000002007650001.png new file mode 100644 index 00000000..8ef8b5d5 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007650001.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007717733.png b/docs/mrs/umn/en-us_image_0000002007717733.png new file mode 100644 index 00000000..25c78ba2 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007717733.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007717737.png b/docs/mrs/umn/en-us_image_0000002007717737.png new file mode 100644 index 00000000..c3059636 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007717737.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007758273.png b/docs/mrs/umn/en-us_image_0000002007758273.png new file mode 100644 index 00000000..6b291213 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007758273.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007758277.png b/docs/mrs/umn/en-us_image_0000002007758277.png new file mode 100644 index 00000000..a9cc467d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007758277.png differ diff --git a/docs/mrs/umn/en-us_image_0000002007968137.png b/docs/mrs/umn/en-us_image_0000002007968137.png new file mode 100644 index 00000000..bac2a387 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002007968137.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008102449.png b/docs/mrs/umn/en-us_image_0000002008102449.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008102449.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008102489.png b/docs/mrs/umn/en-us_image_0000002008102489.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008102489.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008102517.png b/docs/mrs/umn/en-us_image_0000002008102517.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008102517.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008128989.png b/docs/mrs/umn/en-us_image_0000002008128989.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008128989.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008129021.png b/docs/mrs/umn/en-us_image_0000002008129021.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008129021.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008129057.png b/docs/mrs/umn/en-us_image_0000002008129057.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008129057.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008129089.png b/docs/mrs/umn/en-us_image_0000002008129089.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008129089.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008129121.png b/docs/mrs/umn/en-us_image_0000002008129121.png new file mode 100644 index 00000000..b678574f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008129121.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008129137.png b/docs/mrs/umn/en-us_image_0000002008129137.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008129137.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008129141.png b/docs/mrs/umn/en-us_image_0000002008129141.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008129141.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008129145.png b/docs/mrs/umn/en-us_image_0000002008129145.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008129145.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008129157.png b/docs/mrs/umn/en-us_image_0000002008129157.png new file mode 100644 index 00000000..56604e56 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008129157.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008129165.png b/docs/mrs/umn/en-us_image_0000002008129165.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008129165.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008222021.png b/docs/mrs/umn/en-us_image_0000002008222021.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008222021.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248417.png b/docs/mrs/umn/en-us_image_0000002008248417.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248417.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248449.png b/docs/mrs/umn/en-us_image_0000002008248449.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248449.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248489.png b/docs/mrs/umn/en-us_image_0000002008248489.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248489.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248521.png b/docs/mrs/umn/en-us_image_0000002008248521.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248521.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248553.png b/docs/mrs/umn/en-us_image_0000002008248553.png new file mode 100644 index 00000000..8bc7f45f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248553.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248581.png b/docs/mrs/umn/en-us_image_0000002008248581.png new file mode 100644 index 00000000..bd8bef94 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248581.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248585.png b/docs/mrs/umn/en-us_image_0000002008248585.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248585.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248597.png b/docs/mrs/umn/en-us_image_0000002008248597.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248597.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248601.png b/docs/mrs/umn/en-us_image_0000002008248601.png new file mode 100644 index 00000000..3995475d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248601.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008248613.png b/docs/mrs/umn/en-us_image_0000002008248613.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008248613.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008258961.png b/docs/mrs/umn/en-us_image_0000002008258961.png new file mode 100644 index 00000000..d6da7802 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008258961.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008258969.png b/docs/mrs/umn/en-us_image_0000002008258969.png new file mode 100644 index 00000000..548b52a8 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008258969.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008258977.png b/docs/mrs/umn/en-us_image_0000002008258977.png new file mode 100644 index 00000000..c6b4acb2 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008258977.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008258989.png b/docs/mrs/umn/en-us_image_0000002008258989.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008258989.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008299541.png b/docs/mrs/umn/en-us_image_0000002008299541.png new file mode 100644 index 00000000..61ba0343 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008299541.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008475541.png b/docs/mrs/umn/en-us_image_0000002008475541.png new file mode 100644 index 00000000..582605f5 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008475541.png differ diff --git a/docs/mrs/umn/en-us_image_0000002008555981.png b/docs/mrs/umn/en-us_image_0000002008555981.png new file mode 100644 index 00000000..6a0422a1 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002008555981.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009454237.png b/docs/mrs/umn/en-us_image_0000002009454237.png new file mode 100644 index 00000000..5e22e61f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009454237.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009454273.png b/docs/mrs/umn/en-us_image_0000002009454273.png new file mode 100644 index 00000000..0affa0b8 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009454273.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009454277.png b/docs/mrs/umn/en-us_image_0000002009454277.png new file mode 100644 index 00000000..1eba0015 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009454277.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009454285.png b/docs/mrs/umn/en-us_image_0000002009454285.png new file mode 100644 index 00000000..cb096026 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009454285.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009454293.png b/docs/mrs/umn/en-us_image_0000002009454293.png new file mode 100644 index 00000000..ea6fded8 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009454293.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009573745.png b/docs/mrs/umn/en-us_image_0000002009573745.png new file mode 100644 index 00000000..6167506a Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009573745.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009573749.png b/docs/mrs/umn/en-us_image_0000002009573749.png new file mode 100644 index 00000000..5694475f Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009573749.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009573789.png b/docs/mrs/umn/en-us_image_0000002009573789.png new file mode 100644 index 00000000..4925e01e Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009573789.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009573793.png b/docs/mrs/umn/en-us_image_0000002009573793.png new file mode 100644 index 00000000..b217677d Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009573793.png differ diff --git a/docs/mrs/umn/en-us_image_0000002009573797.png b/docs/mrs/umn/en-us_image_0000002009573797.png new file mode 100644 index 00000000..1eba0015 Binary files /dev/null and b/docs/mrs/umn/en-us_image_0000002009573797.png differ diff --git a/docs/mrs/umn/en-us_topic_0012808230.html b/docs/mrs/umn/en-us_topic_0012808230.html index 8d83fb2b..eaafff60 100644 --- a/docs/mrs/umn/en-us_topic_0012808230.html +++ b/docs/mrs/umn/en-us_topic_0012808230.html @@ -38,7 +38,7 @@++Parent topic: SQL Inspector+The cluster scale-in progress includes:-- Preparing for scale-in
- Decommissioning instance
- Deleting VMs
- Deleting nodes from the cluster
- Scale-in failed
The system will display causes of cluster installation, scale-out, and scale-in failures. For details, see Table 5.
+The system will display causes of cluster installation, scale-out, and scale-in failures. For details, see Table 6.
- OLdap port number obtaining method:
- Use the following method to confirm whether the internal system username is locked:
User Account ListUser Account ListSystem User Description
+User Account List
-User Classification
The MRS cluster provides the following three types of users. The system administrator needs to periodically change the passwords. It is not recommended to use the default passwords.
@@ -192,7 +192,7 @@ This section describes the default users in the MRS cluster.
This user is the IoTDB system administrator and has the following user permissions:
If the current cluster is not the cluster that is installed for the first time in MRS Manager, the default user group name of all components except Manager in the cluster is c<cluster ID>_ default user group name, for example, c2_hadoop.
User
For details, see User Account ListUser Account ListSystem User Description.
+User
For details, see User Account List.
Service-related User Security Parameters
Created
diff --git a/docs/mrs/umn/mrs_01_0027.html b/docs/mrs/umn/mrs_01_0027.html index 0ee9dfdc..ba641cf5 100644 --- a/docs/mrs/umn/mrs_01_0027.html +++ b/docs/mrs/umn/mrs_01_0027.html @@ -4,11 +4,12 @@The first step of using MRS is to create a cluster. This section describes how to create a cluster on the MRS management console.
Procedure
- Log in to the MRS console.
- Click Create Cluster, the Create Cluster page is displayed-
When creating a cluster, pay attention to quota notification. If a resource quota is insufficient, increase the resource quota as prompted and create a cluster.
- On the page for create a cluster, click the Custom Config tab.
- Configure cluster software information.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
- Cluster Version: Select the latest version, which is the default value.
- Cluster Type: Use the default Analysis Cluster.
- Component Port: Use the default Open source.
- Component: Select components such as Spark2x, HBase, and Hive for the analysis cluster. For a streaming cluster, select components such as Kafka and Storm. For a hybrid cluster, you can select the components of the analysis cluster and streaming cluster based on service requirements.
- Click Next.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- Security Group: Select Auto create.
- EIP: Select Bind later.
- Enterprise Project: Use the default value.
- Instance Specifications: Select General Computing S3 -> 8 vCPUs | 16 GB (s3.2xlarge.2) for both Master and Core nodes.
- System Disk: Select Common I/O and retain the default settings.
- Data Disk: Select Common I/O and retain the default settings.
- Instance Count: The default number of Master nodes is 2, and that of Core nodes is 3.
- Click Next. The Set Advanced Options tab page is displayed. Configure the following parameters. Retain the default settings for the other parameters.
- Kerberos authentication:
- Kerberos Authentication: Disable Kerberos authentication.
- Username: name of the Manager administrator. admin is used by default.
- Password: password of the Manager administrator.
- Login Mode: Select a mode for logging in to an ECS.
- Password: Set a password for logging in to an ECS.
- Key Pair: Select a key pair from the drop-down list. Select "I acknowledge that I have obtained private key file SSHkey-xxx and that without this file I will not be able to log in to my ECS." If you have never created a key pair, click View Key Pair to create or import a key pair. And then, obtain a private key file.
- Secure Communications: Select Enable.
- Kerberos authentication:
- On the page for create a cluster, click the Custom Config tab.
- Configure cluster software information.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
- Cluster Type: Use the default Analysis Cluster.
- Cluster Version: Select the latest version, which is the default value.
- Component: Select components such as Spark2x, HBase, and Hive for the analysis cluster. For a streaming cluster, select components such as Kafka and Storm. For a hybrid cluster, you can select the components of the analysis cluster and streaming cluster based on service requirements.
- Component Port: Use the default Open source.
- Click Next.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- Security Group: Select Auto create.
- EIP: Select Bind later.
- CPU Architecture: Use the default value.
- Common Template: Select a template based on service requirements.
- Cluster Nodes
- Node Count: the number of nodes you want to purchase. For MRS 3.x clusters, the default value is 3. You can set the value as you need.
- Instance Specifications: Retain the default settings for master and core nodes or select proper specifications based on service requirements.
- System Disk: Retain the default Ultra-high I/O and storage capacity.
- Data Disk: Retain the default Ultra-high I/O, storage capacity, and quantity.
- Topology Adjustment: Retain the default value.
- Click Next. The Set Advanced Options tab page is displayed. Configure the following parameters. Retain the default settings for the other parameters.
- Kerberos Authentication: Disable Kerberos authentication.
- Username: name of the Manager administrator. admin is used by default.
- Password: password of the Manager administrator.
- Confirm Password: Enter the password again.
- Login Mode: Select a mode for logging in to an ECS.
- Password: Set a password for logging in to an ECS.
- Key Pair: Select a key pair from the drop-down list. Select "I acknowledge that I have obtained private key file SSHkey-xxx and that without this file I will not be able to log in to my ECS." If you have never created a key pair, click View Key Pair to create or import a key pair. And then, obtain a private key file.
- Set Advanced Options: To configure some advanced parameters, select Configure.
- Secure Communications: Select Enable.
- Click Next.
- Configure: Confirm the parameters configured in the Configure Software, Configure Hardware, and Set Advanced Options areas.
- Select the check box for Secure Communications.
- Click Apply Now.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
- Click Back to Cluster List to view the cluster status.
It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.
- Creating a Custom Topology Cluster
-
- Adding a Tag to a Cluster
+- Adding a Tag to a Cluster/Node
- Communication Security Authorization
diff --git a/docs/mrs/umn/mrs_01_0048.html b/docs/mrs/umn/mrs_01_0048.html index e6e4ccbd..7802f2b7 100644 --- a/docs/mrs/umn/mrs_01_0048.html +++ b/docs/mrs/umn/mrs_01_0048.html @@ -1,52 +1,61 @@ -
Adding a Tag to a Cluster
-Tags are used to identify clusters. Adding tags to clusters can help you identify and manage your cluster resources.
-You can add a maximum of 10 tags to a cluster when creating the cluster or add them on the details page of the created cluster.
+Adding a Tag to a Cluster/Node
+Tags are used to identify clusters/nodes. Adding tags to clusters can help you identify and manage your cluster resources.
+- Cluster tags: You can add up to 10 tags to a cluster during cluster creation or add them on the details page of a created cluster. Updating a cluster tag will synchronize the tag to all nodes in the cluster.
- Node tags: You can use the default tag or add tags to nodes in an MRS cluster when you configure an auto scaling policy. Node tags take the tag quotas. You can view the tags of a node in the Nodes tab on MRS console.
- Default tags: An MRS cluster contains multiple nodes, and each node is an ECS and contains EVS disks. After the default tag is enabled, the system automatically creates a cluster tag and a tag for each node. The default tag is automatically synchronized to the corresponding ECS or EVS instances.
To view node tags, go to the Nodes tab on the MRS console, and move the cursor to the tag icon of a node in the node list.
++ - MRS tag updates are synchronized to the ECSs or EVS disks in the cluster. However, if you modify MRS cluster tags on the ECS or EVS console, the modification will not be synchronized to MRS. To ensure tag consistency, do not modify MRS cluster tags on the ECS or EVS console.
- You can add a maximum of 10 tags to a cluster. If the number of tags of a node in the cluster reaches the upper limit, no more tags can be added to the cluster.
- If default tags are enabled, a default tag is added to the cluster and each node, which takes two quotas. That is, a maximum of 10 tags can be added by default. In this case, a maximum of eight more tags can still be added.
If your organization has configured tag strategies for MRS, add tags to clusters/nodes based on the strategies. If a tag does not comply with the tag strategies, the cluster/node may be failed to be created. Contact the organization administrator to learn more about the tag strategies.
A tag consists of a tag key and a tag value. Table 1 provides tag key and value requirements.
--Table 1 Tag key and value requirements Parameter
+-Table 1 Tag key and value requirements Parameter
Requirement
+Requirement
Example
+Example
Key
+Key
A tag key cannot be left blank.
+A tag key cannot be left blank.
A tag key must be unique in a cluster.
A tag key contains a maximum of 36 characters.
A tag value cannot contain special characters (=*<>\,|/) or start or end with spaces.
Organization
+Organization
Value
+Value
A tag value contains a maximum of 43 characters.
+A tag value contains a maximum of 43 characters.
A tag value cannot contain special characters (=*<>\,|/) or start or end with spaces. This parameter can be left blank.
Apache
+Apache
Adding Tags to a Cluster
You can perform the following operations to add tags to a cluster when creating the cluster.- Log in to the MRS console.
- Click Create Cluster. The corresponding page is displayed.
- Click the Custom Config tab.
- Configure the cluster software and hardware by referring to Creating a Custom Cluster.
- On the Set Advanced Options tab page, add a tag.
Enter the key and value of a tag to be added.
-You can add a maximum of 10 tags to a cluster and use intersections of tags to search for the target cluster.
- You can also add tags to existing clusters. For details, see Managing Tags.
++Adding Tags to a Cluster
- Adding cluster tags during cluster creation
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Custom Config tab.
- Configure the cluster software and hardware by referring to Creating a Custom Cluster.
- Select Configure on the right of Set Advanced Options and enter the key and value of a new tag.
- Adding tags to an existing cluster
- Log in to the MRS management console.
- In the navigation pane on the left, choose . Click the name of a running cluster. The basic information page of the cluster is displayed.
- Click the Tags tab.
- Click Add/Edit Tag. If this is your first time adding a tag, click Add Tag. In the displayed dialog box, enter the key and value of a tag, and click Add.-
You can also add cluster tags by enabling default tags. All nodes will be tagged with the cluster ID and node IDs, which takes two quotas.
- Click OK. +
Adding Tags to a Node
- Node tags are automatically added when a default tag is added to a cluster. For details, see Adding tags to an existing cluster.
- Adding node tags for auto scaling
If you add a tag when configuring an auto scaling policy, MRS automatically adds the tag to the new nodes and synchronizes the tag to the ECSs and EVS disks.
+- Log in to the MRS management console.
- In the navigation pane on the left, choose . Click the name of a running cluster. The basic information page of the cluster is displayed.
- On the page that is displayed, click the Auto Scaling tab.
- Click Edit on the right of an existing auto scaling policy. In the displayed dialog box, enter the key and value of the tag you want to add, and click Add.+
- You need to enable the auto scaling policy and configure scale-out rules. Otherwise, the node tags will not take effect.
- If tag quotas are insufficient, delete the cluster tag or modify existing a tag of the auto scaling policy, and then enable the policy.
- Tags cannot be added to auto scaling policies of resource pools.
- Click OK.
-Searching for the Target Cluster
On the Active Clusters page, search for the target cluster by tag key or tag value.- Log in to the MRS console.
- In the upper right corner of the Active Clusters page, click Search by Tag to access the search page.
- Enter the tag of the cluster to be searched.
You can select a tag key or tag value from their drop-down lists. When the tag key or tag value is exactly matched, the system can automatically locate the target cluster. If you enter multiple tags, their intersections are used to search for the cluster.
- Click Search.
The system searches for the target cluster by tag key or value.
Managing Tags
You can view, add, modify, and delete tags on the Tags tab page of the cluster.- Log in to the MRS console.
- On the Active Clusters page, click the name of a cluster for which you want to manage tags.
The cluster details page is displayed.
+Managing Tags
You can view, add, modify, and delete tags on the Tags tab page of the cluster.- Log in to the MRS console.
- On the Active Clusters page, click the name of a cluster for which you want to manage tags.
The cluster details page is displayed.
- Click the Tags tab and view, add, modify, and delete tags on the tab page.
- View
On the Tags tab page, you can view details about tags of the cluster, including the number of tags and the key and value of each tag.
- Add
Click Add Tag in the upper left corner. In the displayed Add Tag dialog box, enter the key and value of the tag to be added, and click OK.
- Modify
In the Operation column of the tag, click Edit. In the displayed Edit Tag page, enter new tag key and value and click OK.
diff --git a/docs/mrs/umn/mrs_01_0121.html b/docs/mrs/umn/mrs_01_0121.html index 00fdbb0e..ebf424f0 100644 --- a/docs/mrs/umn/mrs_01_0121.html +++ b/docs/mrs/umn/mrs_01_0121.html @@ -6,9 +6,11 @@- Separated deployment of the management and control roles: The management role and control role are deployed on different Master nodes.
- Co-deployment of the management and control roles: The management and control roles are co-deployed on the Master node.
- ZooKeeper is deployed on an independent node to improve reliability.
- Components are deployed separately to avoid resource contention.
Roles in an MRS cluster:
- Management Node (MN): is the node to install Manager (the management system of the MRS cluster). It provides a unified access entry. Manager centrally manages nodes and services deployed in the cluster.
- Control Node (CN): controls and monitors how data nodes store and receive data, and send process status, and provides other public functions. Control nodes of MRS include HMaster, HiveServer, ResourceManager, NameNode, JournalNode, and SlapdServer.
- Data Node (DN): A data node executes the instructions sent by the management node, reports task status, stores data, and provides other public functions. Data nodes of MRS include DataNode, RegionServer, and NodeManager.
Customizing a Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Custom Config tab.
- Configure basic cluster information. For details about the parameters, see Software Configurations.
- Region: Retain the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
- Cluster Version: Currently, only MRS 3.x are supported.
- Click Next. Configure hardware information.
- AZ: Retain the default value.
- VPC: Retain the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Retain the default value.
- Security Group: Select Auto create.
- EIP: Select Bind later.
- Enterprise Project: Retain the default value.
- Common Node: For details, see Custom Cluster Template Description.
- Instance Specifications: Click
to configure the instance specifications, system disk and data disk storage types, and storage space. - Instance Count: Adjust the number of cluster instances based on the service volume. For details, see Table 2.
- Topology Adjustment: If the deployment mode in the Common Node does not meet the requirements, you need to manually install some instances that are not deployed by default, or you need to manually install some instances, set Topology Adjustment to Enable and adjust the instance deployment mode based on service requirements. For details, see Topology Adjustment for a Custom Cluster.
- Click Next and set advanced options.
For details about the parameters, see (Optional) Advanced Configuration.
+Customizing a Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Custom Config tab.
- Configure basic cluster information. For details about the parameters, see Software Configurations.
- Region: Retain the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
- Cluster Type: Use the default value.
- Cluster Version: Select the latest version, which is the default value.
- Component: Select components such as Spark2x, HBase, and Hive for the analysis cluster. For a streaming cluster, select components such as Kafka and Storm. For a hybrid cluster, you can select the components of the analysis cluster and streaming cluster based on service requirements.
- Component Port: Use the default Open source.
- Click Next. Configure hardware information.
- AZ: Retain the default value.
- VPC: Retain the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Retain the default value.
- Security Group: Select Auto create.
- EIP: Select Bind later.
- CPU Architecture: Retain the default value.
- Common Template: Select a template based on service requirements.
- Cluster Nodes
- Node Count: the number of nodes you want to purchase. For MRS 3.x clusters, the default value is 3. You can set the value as you need.
- Instance Specifications: Retain the default settings for master and core nodes or select proper specifications based on service requirements.
- System Disk: Retain the default Ultra-high I/O and storage capacity.
- Data Disk: Retain the default Ultra-high I/O, storage capacity, and quantity.
- Topology Adjustment: If the deployment mode in the Common Node does not meet the requirements, you need to manually install some instances that are not deployed by default, or you need to manually install some instances, set Topology Adjustment to Enable and adjust the instance deployment mode based on service requirements. For details, see Topology Adjustment for a Custom Cluster.
- Click Next and set advanced options.
For details about the parameters, see Advanced Options.
+ - Click Next.
- Configure: Confirm the parameters configured in the Configure Software, Configure Hardware, and Set Advanced Options areas.
- Select the check box for Secure Communications.
- Click Create Now.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
- Click Back to Cluster List to view the cluster status.
It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.
- View
- Adding cluster tags during cluster creation
Table 2 Node deployment scheme of a customized MRS cluster Node Deployment Principle
+diff --git a/docs/mrs/umn/mrs_01_0340.html b/docs/mrs/umn/mrs_01_0340.html index 0f204130..ed22ca35 100644 --- a/docs/mrs/umn/mrs_01_0340.html +++ b/docs/mrs/umn/mrs_01_0340.html @@ -34,8 +34,6 @@Table 2 Node deployment scheme of a customized MRS cluster Node Deployment Principle
Applicable Scenario
CDL
-(applicable only to MRS 3.2.0 and later versions)
+(applicable only to MRS 3.2.0)
- Depends on Kafka.
- Depends on DBService.
Hadoop
- - - - - -Depends on ZooKeeper.
HBase
- - - -Depends on Hadoop.
@@ -572,7 +564,7 @@IoTDB
-(applicable only to MRS 3.2.0 and later versions)
+(applicable only to MRS 3.2.0)
Depends on KerbServer.
-
JobGateway
+- Depends on Hadoop.
- Depends on DBService.
JS(JobServer)
+This role can only be deployed in the master node group.
+The number of role instances must be at least 2 and at most 10.
+-
+JB(JobBalancer)
+This role can be deployed in all node groups.
+Number of role instances to be deployed: 2
+-
+Guardian
+- Depends on Hadoop.
- Depends on ZooKeeper.
TS(TokenServer)
+This role can be deployed in all node groups.
+The number of role instances must be at least 2 and at most 100.
+The Guardian component needs to be installed only when OBS is connected.
+Doris
+Doris depends on LdapServer.
+FE
+You can deploy the role on 1 to 199 nodes. You can set the number with a step size of 2.
+When the FE native port is used to create a cluster, it conflicts with Yarn ResourceManager native port 8030. Do not deploy FE and Yarn ResourceManager on the same node.
+BE
+This role can be deployed in all node groups.
+The number of role instances must be at least 3 and at most 200.
+-
+DBroker
+This role can be deployed in all node groups.
+Deployed role instances must be no more than 200.
+-
+DBalancer
+Optional. This role can be deployed in all node groups.
+The number of role instances must be at least 2 and at most 9.
+-
+- Configuring Users to Access Resources of a Trusted Cluster
-
- Configuring Fine-Grained Permissions for MRS Multi-User Access to OBS
-diff --git a/docs/mrs/umn/mrs_01_0417.html b/docs/mrs/umn/mrs_01_0417.html index f2c6dcfb..bad65cc2 100644 --- a/docs/mrs/umn/mrs_01_0417.html +++ b/docs/mrs/umn/mrs_01_0417.html @@ -9,7 +9,7 @@ source /opt/Bigdata/client/bigdata_env;hadoop fs -D fs.obs.endpoint=<obs-endpoint> -D fs.obs.access.key=<your-ak> -D fs.obs.secret.key=<your-sk> -copyToLocal obs://yourbucket/myfile.tar.gz ./ mkdir -p /<your-dir> tar -zxvf myfile.tar.gz -C /<your-dir> - - In MRS 3.x and later versions, the default installation path of the client is /opt/Bigdata/client. In MRS 3.x and earlier versions, the default installation path is /opt/client. For details, see the actual situation.
- The Hadoop client has been preinstalled on the MRS node. You can run the hadoop fs command to download or upload data from or to OBS.
- Obtain the obs-endpoint of each region. For details, see Regions and Endpoints.
- Sample Scripts shows that the installation packages have been uploaded to the public readable OBS file system. Therefore, you can run the curl command in the sample script to download the installation packages.
- In MRS 3.x and later versions, the default installation path of the client is /opt/Bigdata/client. In MRS 3.x and earlier versions, the default installation path is /opt/client. For details, see the actual situation.
- The Hadoop client has been preinstalled on the MRS node. You can run the hadoop fs command to download or upload data from or to OBS.
- Obtain the obs-endpoint of each region.
- Sample Scripts shows that the installation packages have been uploaded to the public readable OBS file system. Therefore, you can run the curl command in the sample script to download the installation packages.
Uploading the Script to the OBS File System
After script compilation, upload the script to the OBS file system in the same region. At the time you specify, each node in the cluster downloads the script from OBS and executes the script as user root.
diff --git a/docs/mrs/umn/mrs_01_0440.html b/docs/mrs/umn/mrs_01_0440.html index b35b3061..9d5eae70 100644 --- a/docs/mrs/umn/mrs_01_0440.html +++ b/docs/mrs/umn/mrs_01_0440.html @@ -4,13 +4,11 @@diff --git a/docs/mrs/umn/mrs_01_0453.html b/docs/mrs/umn/mrs_01_0453.html index 0f3a2cab..72407627 100644 --- a/docs/mrs/umn/mrs_01_0453.html +++ b/docs/mrs/umn/mrs_01_0453.html @@ -5,8 +5,6 @@-
-
- Introduction to Storage-Compute Decoupling
+ - MRS Storage-Compute Decoupling
- - Configuring a Storage-Compute Decoupled Cluster (Agency)
+ - Interconnecting with OBS Using the Cluster Agency Mechanism
- - Configuring a Storage-Compute Decoupled Cluster (AK/SK)
-
- - Using a Storage-Compute Decoupled Cluster
+ - Interconnecting with OBS Using the Guardian Service
- Create IAM users under your cloud account for employees based on your enterprise's organizational structure so that each employee is allowed to access MRS resources using their unique security credential (IAM user).
- Grant only the permissions required for users to perform a specific task.
- Entrust a cloud account or cloud service to perform efficient O&M on your MRS resources.
If your cloud account does not require IAM users, skip this section.
This section describes the procedure for granting permissions (see Figure 1).
-Prerequisites
Learn about the permissions. For the permissions of other services, see Permission Description.
-Process Flow
- Create a user group and assign permissions to it.
Create a user group on the IAM console, and assign MRS permissions to the group.
- Create a user and add it to a user group.
Create a user on the IAM console and add the user to the group created in 1. Create a user group and assign permissions to it.
@@ -19,41 +17,41 @@You can grant users permissions by using roles and policies.
- Roles: A type of coarse-grained authorization mechanism that defines permissions related to user responsibilities. This mechanism provides only a limited number of service-level roles for authorization. When using roles to grant permissions, you need to also assign other roles on which the permissions depend to take effect. However, roles are not an ideal choice for fine-grained authorization and secure access control.
- Policies: A type of fine-grained authorization mechanism that defines permissions required to perform operations on specific cloud resources under certain conditions. This mechanism allows for more flexible policy-based authorization, meeting requirements for secure access control. For example, you can grant MRS users only the permissions for performing specified operations on MRS clusters, such as creating a cluster and querying a cluster list rather than deleting a cluster. Most policies define permissions based on APIs.
Table 1 lists all the system policies supported by MRS. -Table 1 MRS system policies Policy
+Table 1 MRS system policies Policy
Description
+Description
Type
+Type
MRS FullAccess
+MRS FullAccess
Administrator permissions for MRS. Users granted these permissions can operate and use all MRS resources.
+Administrator permissions for MRS. Users granted these permissions can operate and use all MRS resources.
Fine-grained policy
+Fine-grained policy
MRS CommonOperations
+MRS CommonOperations
Common user permissions for MRS. Users granted these permissions can use MRS but cannot add or delete resources.
+Common user permissions for MRS. Users granted these permissions can use MRS but cannot add or delete resources.
Fine-grained policy
+Fine-grained policy
MRS ReadOnlyAccess
+MRS ReadOnlyAccess
Read-only permission for MRS. Users granted these permissions can only view MRS resources.
+Read-only permission for MRS. Users granted these permissions can only view MRS resources.
Fine-grained policy
+Fine-grained policy
MRS Administrator
+MRS Administrator
Permissions:
+Permissions:
- All operations on MRS
- Users with permissions of this policy must also be granted permissions of the Tenant Guest and Server Administrator policies.
RBAC policy
+RBAC policy
Table 2 lists the common operations supported by each system-defined policy or role of MRS. Select the policies or roles as required.
--Table 2 Common operations supported by each system-defined policy Operation
+@@ -123,7 +122,13 @@Table 2 Common operations supported by each system-defined policy Operation
MRS FullAccess
+MRS FullAccess
MRS CommonOperations
+MRS CommonOperations
MRS ReadOnlyAccess
+MRS ReadOnlyAccess
MRS Administrator
+MRS Administrator
Creating a cluster
+Creating a cluster
√
+√
x
+x
x
+x
√
+√
Resizing a cluster
+Resizing a cluster
√
+√
x
+x
x
+x
√
+√
Upgrading node specifications
+Upgrading node specifications
√
+√
x
+x
x
+x
√
+√
Deleting a cluster
+Deleting a cluster
√
+√
x
+x
x
+x
√
+√
Querying cluster details
+Querying cluster details
√
+√
√
+√
√
+√
√
+√
Querying a cluster list
+Querying a cluster list
√
+√
√
+√
√
+√
√
+√
Configuring an auto scaling rule
+Configuring an auto scaling rule
√
+√
x
+x
x
+x
√
+√
Querying a host list
+Querying a host list
√
+√
√
+√
√
+√
√
+√
Querying operation logs
+Querying operation logs
√
+√
√
+√
√
+√
√
+√
Creating and executing a job
+Creating and executing a job
√
+√
√
+√
x
+x
√
+√
Stopping a job
+Stopping a job
√
+√
√
+√
x
+x
√
+√
Deleting a single job
+Deleting a single job
√
+√
√
+√
x
+x
√
+√
Deleting jobs in batches
+Deleting jobs in batches
√
+√
√
+√
x
+x
√
+√
Querying job details
+Querying job details
√
+√
√
+√
√
+√
√
+√
Querying a job list
+Querying a job list
√
+√
√
+√
√
+√
√
+√
Creating a folder
+Creating a folder
√
+√
√
+√
x
+x
√
+√
Deleting a file
+Deleting a file
√
+√
√
+√
x
+x
√
+√
Querying a file list
+Querying a file list
√
+√
√
+√
√
+√
√
+√
Operating cluster tags in batches
+Operating cluster tags in batches
√
+√
√
+√
x
+x
√
+√
Creating a single cluster tag
+Creating a single cluster tag
√
+√
√
+√
x
+x
√
+√
Deleting a single cluster tag
+Deleting a single cluster tag
√
+√
√
+√
x
+x
√
+√
Querying a resource list by tag
+Querying a resource list by tag
√
+√
√
+√
√
+√
√
+√
Querying cluster tags
+Querying cluster tags
√
+√
√
+√
√
+√
√
+√
Accessing Manager
+Accessing Manager
√
+√
√
+√
x
+x
√
+√
Querying a patch list
+Querying a patch list
√
+√
√
+√
√
+√
√
+√
Installing a patch
+Installing a patch
√
+√
√
+√
x
+x
√
+√
Uninstalling a patch
+Uninstalling a patch
√
+√
√
+√
x
+x
√
+√
Authorizing O&M channels
+Authorizing O&M channels
√
+√
√
+√
x
+x
√
+√
Sharing O&M channel logs
+Sharing O&M channel logs
√
+√
√
+√
x
+x
√
+√
Querying an alarm list
+Querying an alarm list
√
+√
√
+√
√
+√
√
+√
Subscribing to alarm notification
+Subscribing to alarm notification
√
+√
√
+√
x
+x
√
+√
Submitting an SQL statement
+Submitting an SQL statement
√
+√
√
+√
x
+x
√
+√
Querying SQL results
+Querying SQL results
√
+√
√
+√
x
+x
√
+√
Canceling an SQL execution task
+Canceling an SQL execution task
√
+√
√
+√
x
+x
√
+√
Introduction to Storage-Compute Decoupling
+MRS Storage-Compute Decoupling
In scenarios that require large storage capacity and elastic compute resources, MRS enables you to store data in OBS and use an MRS cluster for data computing only. In this way, storage and compute are separated.
In the big data decoupled storage-compute scenario, the OBS parallel file system must be used to configure a cluster. Using common object buckets will greatly affect the cluster performance.
Process of using the storage-compute decoupling function:
-- Configure a storage-compute decoupled cluster using either of the following methods (agency is recommended):
- Bind an agency of the ECS type to an MRS cluster to access OBS, preventing the AK/SK from being exposed in the configuration file. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).
- Configure the AK/SK in an MRS cluster. The AK/SK will be exposed in the configuration file in plaintext. Exercise caution when performing this operation. For details, see Configuring a Storage-Compute Decoupled Cluster (AK/SK).
- Use the cluster.
+
- Configure a storage-compute decoupled cluster using either of the following methods (agency is recommended):
- Bind an agency of the ECS type to an MRS cluster to access OBS, preventing the AK/SK from being exposed in the configuration file. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).
- Configure the AK/SK in an MRS cluster. The AK/SK will be exposed in the configuration file in plaintext. Exercise caution when performing this operation. For details, see Configuring a Storage-Compute Decoupled Cluster (AK/SK).
- MRS uses the Guardian component to connect to OBS, providing other components with the capabilities of obtaining temporary authentication credentials and fine-grained permission control for accessing OBS. For details, see Interconnecting the Guardian Service with OBS.+
- Currently, using Guardian components to connect to OBS is supported only in MRS 3.3.0-LTS or later versions. For details about configurations in clusters of other versions, see Interconnecting MRS with OBS Using an Agency.
- Job submission based on the Guardian storage and compute decoupling management plane depends on JobGateway instead of Executor.
- Use the cluster.
After the required permissions for accessing OBS are obtained, components in the MRS cluster can access the corresponding files through the client.
+For details about how to configure components to access OBS, see the following content:
+
- Configure a storage-compute decoupled cluster using either of the following methods (agency is recommended):
diff --git a/docs/mrs/umn/mrs_01_0468.html b/docs/mrs/umn/mrs_01_0468.html index 5fc217d3..abf93bcc 100644 --- a/docs/mrs/umn/mrs_01_0468.html +++ b/docs/mrs/umn/mrs_01_0468.html @@ -4,7 +4,9 @@In MRS 1.9.2 or later, OBS can be interconnected with MRS using obs://. Currently, Hadoop, Hive, Spark, Presto, and Flink are supported. HBase cannot use obs:// to interconnect with OBS.
MRS provides the following configuration modes for accessing OBS. You can select one of them. The agency mode is recommended.
- Bind an agency of the ECS type to an MRS cluster to access OBS, preventing the AK/SK from being exposed in the configuration file. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).
- Configure the AK/SK in an MRS cluster. The AK/SK will be exposed in the configuration file in plaintext. Exercise caution when performing this operation. For details, see the following part in this section.
- To improve data write performance, change the value of the fs.obs.buffer.dir parameter of the corresponding service to a data disk directory.
- In the big data decoupled storage-compute scenario, the OBS parallel file system must be used to configure a cluster. Using common object buckets will greatly affect the cluster performance.
- To improve data write performance, log in to the Manager and choose Cluster > Services > Name of the service to be modified > Configurations. Change the value of fs.obs.buffer.dir to the data disk directory.
- In the storage-compute decoupled scenario, the OBS parallel file system must be used to configure a cluster. Using common object buckets will greatly affect the cluster performance.
- In MRS 3.2.0-LTS.1 and later versions, components prevent mis-deletion by default. That is, file data deleted by component users is not directly deleted but stored in the recycle bin directory in the OBS file system.
To save OBS space, you need to enable periodical deletion of file data from the OBS recycle bin by referring to Configuring the Policy for Clearing Component Data in the Recycle Bin.
+
- Configuration files containing authentication passwords pose security risks. Delete such files after configuration or store them securely.
- Commands carrying authentication passwords pose security risks. Disable historical command recording before running such commands to prevent information leakage.
Using Hadoop to Access OBS
- Add the following content to file core-site.xml in the HDFS directory ($client_home/ HDFS/hadoop/etc/hadoop) on the MRS client:
<property> <name>fs.obs.access.key</name> @@ -81,7 +83,7 @@ fs.obs.endpoint: obs endpoint
diff --git a/docs/mrs/umn/mrs_01_0496.html b/docs/mrs/umn/mrs_01_0496.html index d8ee40c5..a9e72a51 100644 --- a/docs/mrs/umn/mrs_01_0496.html +++ b/docs/mrs/umn/mrs_01_0496.html @@ -2,7 +2,7 @@-Parent topic: Configuring a Cluster with Storage and Compute Decoupled+Quick Creation of an HBase Analysis Cluster
This section describes how to quickly create an HBase query cluster. The HBase cluster uses Hadoop and HBase components to provide a column-oriented distributed cloud storage system featuring enhanced reliability, excellent performance, and elastic scalability. It applies to the storage and distributed computing of massive amounts of data. You can use HBase to build a storage system capable of storing TB- or even PB-level data. With HBase, you can filter and analyze data with ease and get responses in milliseconds, rapidly mining data value.
-Quick Creation of an HBase Analysis Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Quick Config tab.
- Configure basic cluster information. For details about the parameters, see Creating a Custom Cluster.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
- Cluster Version: Select the latest version, which is the default value. (The components provided by a cluster vary according to the cluster version. Select a cluster version based on site requirements.)
- Component: Select HBase Query Cluster.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- Enterprise Project: Use the default value.
- Cluster Node: Select the number of cluster nodes and node specifications based on site requirements. For MRS 3.x or later, the memory of the master node must be greater than 64 GB.
- Cluster HA: Use the default value. This parameter is not available in MRS 3.x.
- Kerberos Authentication: Select whether to enable Kerberos authentication.
- Username: The default value is root/admin. User root is used to remotely log in to ECSs, and user admin is used to access the cluster management page.
- Password: Set a password for user root/admin.
- Confirm Password: Enter the password of user root/admin again.
Quick Creation of an HBase Analysis Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Quick Config tab.
- Configure basic cluster information. For details about the parameters, see Creating a Custom Cluster.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
- Cluster Type: Use the default value.
- Cluster Version: Select the latest version, which is the default value. (The components provided by a cluster vary according to the cluster version. Select a cluster version based on site requirements.)
- Component: Select HBase Query Cluster.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- Enterprise Project: Use the default value.
- CPU Architecture: Use the default value.
- Cluster Nodes: Select the number of cluster nodes and node specifications based on site requirements. For MRS 3.x or later, the memory of the master node must be greater than 64 GB.
- Kerberos Authentication: Select whether to enable Kerberos authentication.
- Username: The default value is root/admin. User root is used to remotely log in to ECSs, and user admin is used to access the cluster management page.
- Password: Set a password for user root/admin.
- Confirm Password: Enter the password of user root/admin again.
- Select Enable to enable secure communications. For details, see Communication Security Authorization.
- Click Create Now.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
- Click Back to Cluster List to view the cluster status. Click Access Cluster to view cluster details.
For details about cluster status during creation, see the description of the status parameters in Table 1.
It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.
diff --git a/docs/mrs/umn/mrs_01_0512.html b/docs/mrs/umn/mrs_01_0512.html index e2ae6de8..5d1a5ada 100644 --- a/docs/mrs/umn/mrs_01_0512.html +++ b/docs/mrs/umn/mrs_01_0512.html @@ -2,7 +2,7 @@Quick Creation of a Hadoop Analysis Cluster
This section describes how to quickly create a Hadoop analysis cluster for analyzing and querying vast amounts of data. In the open-source Hadoop ecosystem, Hadoop uses Yarn to manage cluster resources, Hive and Spark to provide offline storage and computing of large-scale distributed data, Spark Streaming and Flink to offer streaming data computing, and Presto to enable interactive queries, Tez to provide a distributed computing framework of directed acyclic graphs (DAGs).
-Quick Creation of a Hadoop Analysis Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Quick Config tab.
- Configure basic cluster information. For details about the parameters, see Creating a Custom Cluster.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
- Cluster Version: Select the latest version, which is the default value. (The components provided by a cluster vary according to the cluster version. Select a cluster version based on site requirements.)
- Component: Select Hadoop analysis cluster.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- Enterprise Project: Use the default value.
- Cluster Node: Select the number of cluster nodes and node specifications based on site requirements. For MRS 3.x or later, the memory of the master node must be greater than 64 GB.
- Cluster HA: Use the default value. This parameter is not available in MRS 3.x.
- Kerberos Authentication: Select whether to enable Kerberos authentication.
- Username: The default value is root/admin. User root is used to remotely log in to ECSs, and user admin is used to access the cluster management page.
- Password: Set a password for user root/admin.
- Confirm Password: Enter the password of user root/admin again.
Quick Creation of a Hadoop Analysis Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Quick Config tab.
- Configure basic cluster information. For details about the parameters, see Creating a Custom Cluster.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, for example, mrs_20180321.
- Cluster Type: Use the default value.
- Cluster Version: Select the latest version, which is the default value. (The components provided by a cluster vary according to the cluster version. Select a cluster version based on site requirements.)
- Component: Select Hadoop analysis cluster.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- CPU Architecture: Use the default value.
- Cluster Nodes: Select the number of cluster nodes and node specifications based on site requirements. For MRS 3.x or later, the memory of the master node must be greater than 64 GB.
- Kerberos Authentication: Select whether to enable Kerberos authentication.
- Username: The default value is root/admin. User root is used to remotely log in to ECSs, and user admin is used to access the cluster management page.
- Password: Set a password for user root/admin.
- Confirm Password: Enter the password of user root/admin again.
- Select Enable to enable secure communications. For details, see Communication Security Authorization.
- Click Create Now.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
- Click Back to Cluster List to view the cluster status. Click Access Cluster to view cluster details.
For details about cluster status during creation, see the description of the status parameters in Table 1.
It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.
diff --git a/docs/mrs/umn/mrs_01_0513.html b/docs/mrs/umn/mrs_01_0513.html index 04060e4d..866ea228 100644 --- a/docs/mrs/umn/mrs_01_0513.html +++ b/docs/mrs/umn/mrs_01_0513.html @@ -5,9 +5,10 @@You can create an IAM user or user group on the IAM management console and grant it specific operation permissions, to perform refined resource management after registering an account. For details, see Creating an MRS User.
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.-
When creating a cluster, pay attention to quota notification. If a resource quota is insufficient, increase the resource quota as prompted and create a cluster.
- Click the Custom Config tab.
- Configure cluster information by referring to Software Configurations and click Next.
- Configure cluster information by referring to Hardware Configurations and click Next.
- Set advanced options by referring to (Optional) Advanced Configuration and click Apply Now.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
+ - Click the Custom Config tab.
- Configure cluster information by referring to Software Configurations and click Next.
- Configure cluster information by referring to Hardware Configurations and click Next.
- Set advanced options by referring to Advanced Options and click Next.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
+ For any doubt about the pricing, click Pricing details in the lower left corner.
- On the Confirm Configuration page, check the cluster configuration information. If you need to adjust the configuration, click
to go to the corresponding tab page and configure parameters again. - Select the checkbox to enable secure communications. For details, see Communication Security Authorization.
- Click Create Now.
If Kerberos authentication is enabled, check whether this function is required. If it is, click Continue. If not, click Back to disable it and then proceed with the subsequent step. This option cannot be changed after you create a cluster.
- Click Back to Cluster List to view the cluster status.
For details about cluster status during creation, see the description of the status parameters in Table 1.
It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.
On the MRS management console, a maximum of 10 clusters can be concurrently created, and a maximum of 100 clusters can be managed.
@@ -32,9 +33,17 @@The default name is mrs_xxxx. xxxx is a random collection of letters and digits.
+Cluster Type
+- Analysis cluster: is used for offline data analysis and provides Hadoop components.
- Streaming cluster: is used for streaming tasks and provides stream processing components.
- Hybrid cluster: is used for both offline data analysis and streaming processing and provides Hadoop components and streaming processing components. You are advised to use a hybrid cluster to perform offline data analysis and streaming processing tasks at the same time.
- Custom: You can adjust the cluster service deployment mode based on service requirements. For details, see Creating a Custom Cluster. (This type is currently available only in MRS 3.x.)
- Doris cluster: is a high-performance and real-time analytical database for report analysis, ad hoc queries, and acceleration of data lake federation queries.
NOTE:+- MRS streaming clusters do not support job and file management functions.
- To install all components in a cluster, select Custom.
Cluster Version
Currently, MRS 3.1.2-LTS.6 , and 3.2.0-LTS.2 are supported. The default value displayed on the UI varies depending on the version.
+Currently, MRS 3.1.2-LTS.6, MRS 3.2.0-LTS.2 and MRS 3.3.0-LTS.1 are supported. The default value displayed on the UI varies depending on the version.
Cluster Type
@@ -47,11 +56,7 @@Components
MRS components are as follows..
-Components of an analysis cluster:-- Presto: open source and distributed SQL query engine
- Hadoop: distributed system architecture
- Spark2x: A fast general-purpose engine for large-scale data processing. It is developed based on the open-source Spark2.x version.
- Hive: data warehouse framework built on Hadoop
- HBase: distributed column-oriented database
- Tez: an application framework which allows for a complex directed-acyclic-graph of tasks for processing data
- Hue: provides the Hadoop UI capability, which enables users to analyze and process Hadoop cluster data on browsers
- Flink: a distributed big data processing engine that can perform stateful computations over both finite and infinite data streams
- Oozie: a Hadoop job scheduling system
- HetuEngine: a distributed SQL query engine for heterogeneous big data sets
- Ranger: a framework to enable, monitor, and manage data security across the Hadoop platform
- ClickHouse: A column database management system (DBMS) for on-line analytical processing (OLAP).
- Kudu: a column-oriented data store
Components of a streaming cluster:+- Kafka: distributed messaging system
- Flume: distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data
- ZooKeeper: a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services
- Ranger: a framework to enable, monitor, and manage data security across the Hadoop platform
MRS cluster components. For details about component versions supported by different versions of MRS clusters, see List of MRS Component Versions.
Component Port
@@ -107,12 +112,6 @@Enterprise Project
-Select the enterprise project to which the cluster belongs. To use an enterprise project, create one on the Enterprise > Project Management page.
-The Enterprise Management console of the enterprise project is designed for resource management. It helps enterprises manage cloud-based personnel, resources, permissions, and finance in a hierarchical manner, such as management of companies, departments, and projects.
-
Common Node Configurations
+CPU Architecture
+CPU architecture supported by MRS.
+- x86: The x86-based CPU architecture uses Complex Instruction Set Computing (CISC). Each instruction can be used to execute low-level hardware operations. The number of instructions is large, and the length of each instruction is different. Therefore, executing such an instruction is complex and time-consuming.
Common Node Configurations
This parameter is valid only when Cluster Type is set to Custom. For details, see Custom Cluster Template Description.
Instance Count
-Number of Master and Core nodes.
-For Master nodes:
-- If Cluster HA is enabled, the number of Master nodes is fixed to 2.
- If Cluster HA is disabled, the number of Master nodes is fixed to 1.
At least one Core node must exist and the total number of Core and Task nodes cannot exceed 500.
-Task: Click
-
to add a Task node. Click 
to modify the instance specifications and disk configuration of a Task node. Click 
to delete the added Task node.NOTE:-- A maximum of 500 Core nodes are supported by default. If more than 500 Core nodes are required, contact technical support.
- A small number of nodes may cause clusters to run slowly while a large number of nodes may be unnecessarily costly. Set an appropriate value based on data to be processed.
Topology Adjustment
If the deployment mode in the Common Node does not meet the requirements, set Topology Adjustment to Enable and adjust the instance deployment mode based on service requirements. For details, see Topology Adjustment for a Custom Cluster. This parameter is valid only when Cluster Type is set to Custom.
@@ -180,98 +174,131 @@(Optional) Advanced Configuration
-Table 4 MRS cluster advanced configuration topology Parameter
+Advanced Options
++ +
+Table 4 MRS cluster advanced configuration topology Parameter
Description
+Description
Tag
+Kerberos Authentication
For details, see Adding a Tag to a Cluster.
+Whether to enable Kerberos authentication when logging in to Manager. This option cannot be changed after you create a cluster.
+
: If Kerberos Authentication is disabled, common users can use all functions of an MRS cluster. You are advised to disable Kerberos authentication in single-user scenarios.
: If Kerberos Authentication is enabled, common users cannot use the file and job management functions of an MRS cluster and cannot view cluster resource usage or the job records for Hadoop and Spark. To use more cluster functions, the users must contact the Manager administrator to assign more permissions. You are advised to enable Kerberos authentication in multi-user scenarios.- Currently, Presto does not support Kerberos authentication.
Hostname Prefix
+Username
Enter the prefix for the computer hostname of an ECS in the cluster.
+Name of the administrator of Manager. admin is used by default.
Auto Scaling
+Password
Auto scaling can be configured only after you specify Task node specifications in the Configure Hardware step. For details about how to configure Task node specifications, see Configuring an Auto Scaling Rule.
+Password of the Manager administrator
+The following requirements must be met:
+- Must contain 8 to 26 characters.
- Must contain at least four of the following:
- Lowercase letters
- Uppercase letters
- Digits
- At least one of the following special characters: `~!@#$%^&*()-_=+|[{}];:',<.>/?
- Cannot be the same as the username or the username spelled backwards.
Password Strength: The colorbar in red, orange, and green indicates weak, medium, and strong password, respectively.
Agency
+Confirm Password
By binding an agency, ECSs or BMSs can manage some of your resources. Determine whether to configure an agency based on the actual service scenario.
-For example, you can configure an agency of the ECS type to automatically obtain the AK/SK to access OBS. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).
-The MRS_ECS_DEFAULT_AGENCY agency has the OBSOperateAccess permission of OBS and the CESFullAccess (for users who have enabled fine-grained policies), CES Administrator, and KMS Administrator permissions in the region where the cluster is located.
+Enter the password of the Manager administrator again.
Alarm
+Login Mode
If the alarm function is enabled, the cluster maintenance personnel can be notified in a timely manner to locate faults when the cluster runs abnormally or the system is faulty.
-Rule Name
-Name of the rule for sending alarm messages. The value can contain only digits, letters, hyphens (-), and underscores (_).
-Topic Name
-Select an existing topic or click Create Topic to create a topic. To deliver messages published to a topic, you need to add a subscriber to the topic. For details, see Adding Subscriptions to a Topic.
-A topic serves as a message sending channel, where publishers and subscribers can interact with each other.
-Logging
-Whether to collect logs when cluster creation fails.
-After the logging function is enabled, system logs and component run logs are automatically collected and saved to the OBS file system in scenarios such as cluster creation failures and scale-out or scale-in failures for O&M personnel to quickly locate faults. The log information is retained for a maximum of seven days.
-Kerberos Authentication
-Whether to enable Kerberos authentication when logging in to Manager.
--
: If Kerberos Authentication is disabled, common users can use all functions of an MRS cluster. You are advised to disable Kerberos authentication in single-user scenarios. 
: If Kerberos Authentication is enabled, common users cannot use the file and job management functions of an MRS cluster and cannot view cluster resource usage or the job records for Hadoop and Spark. To use more cluster functions, the users must contact the Manager administrator to assign more permissions. You are advised to enable Kerberos authentication in multi-user scenarios.
Username
-Name of the administrator of Manager. admin is used by default.
-Password
-Password of the Manager administrator
-The following requirements must be met:
-- Must contain 8 to 26 characters.
- Must contain at least four of the following:
- Lowercase letters
- Uppercase letters
- Digits
- Have at least one of the following special characters: !?,.: -_{} [ ]@ $% ^ + = /
- Cannot be the same as the username or the username spelled backwards.
Password Strength: The colorbar in red, orange, and green indicates weak, medium, and strong password, respectively.
-Confirm Password
-Enter the password of the Manager administrator again.
-Login Mode
-- Password
You can log in to ECS nodes using a password.
-A password must meet the following requirements:
-- Must be a string and 8 to 26 characters long.
- The password must contain at least four types of the following characters: uppercase letters, lowercase letters, digits, and special characters (! ?,.: -_{} [ ]@ $% ^ + = /).
- The password cannot be the username or the reverse username.
- Key Pair
Key pairs are used to log in to ECS nodes of the cluster. Select a key pair from the drop-down list. Select "I acknowledge that I have obtained private key file SSHkey-xxx and that without this file I will not be able to log in to my ECS." If you have never created a key pair, click View Key Pair to create or import a key pair. And then, obtain a private key file.
-A key pair, also called an SSH key, consists of a public key and a private key. You can create an SSH key and download the private key for authenticating remote login. For security, a private key can only be downloaded once. Keep it secure.
-Use an SSH key in either of the following two methods:
-- Creating an SSH key: After you create an SSH key, a public key and a private key are generated. The public key is stored in the system, and the private key is stored in the local ECS. When you log in to an ECS, the public and private keys are used for authentication.
- Importing an SSH key: If you have obtained the public and private keys, import the public key into the system. When you log in to an ECS, the public and private keys are used for authentication.
- Password
Log in to the ECS or BMS as user root. Enter the password of user root and confirm the password.
+A password must meet the following requirements:
+- Must be a string and 8 to 26 characters long.
- Must contain at least four of the following: uppercase letters, lowercase letters, digits, and special characters (`~!@#$%^&*()-_=+|[{}];:',<.>/?).
- The password cannot be the username or the reverse username.
- Key Pair
Key pairs are used to log in to ECS nodes of the cluster. Select a key pair from the drop-down list. Select "I acknowledge that I have obtained private key file SSHkey-xxx and that without this file I will not be able to log in to my ECS." If you have never created a key pair, click View Key Pair to create or import a key pair. And then, obtain a private key file.
+A key pair, also called an SSH key, consists of a public key and a private key. You can create an SSH key and download the private key for authenticating remote login. For security, a private key can only be downloaded once. Keep it secure.
+Use an SSH key in either of the following two methods:
+- Creating an SSH key: After you create an SSH key, a public key and a private key are generated. The public key is stored in the system, and the private key is stored in the local ECS. When you log in to an ECS, the public and private keys are used for authentication.
- Importing an SSH key: If you have obtained the public and private keys, import the public key into the system. When you log in to an ECS, the public and private keys are used for authentication.
Secure Communications
+Hostname Prefix
MRS clusters provision, manage, and use big data components through the management console. Big data components are deployed in a user's VPC. If the MRS management console needs to directly access big data components deployed in the user's VPC, you need to enable the corresponding security group rules after you have obtained user authorization. This authorization process is called secure communications. For details, see Communication Security Authorization.
-If the secure communications function is not enabled, MRS clusters cannot be created.
+Enter the prefix for the computer hostname of an ECS or BMS in the cluster.
+Setting Advanced Options
+Advanced function parameters of an MRS cluster. Select Configure. For details, see Table 5.
+Table 5 (Optional) Advanced configuration information of the MRS cluster Parameter
+Description
+Tag
+For details, see Adding a Tag to a Cluster/Node.
+Auto Scaling
+Auto scaling can be configured only after you specify task node specifications in the Configure Hardware step by referring to Configuring an Auto Scaling Rule.
+Bootstrap Action
+For details, see Adding a Bootstrap Action.
+Agency
+By binding an agency, ECSs or BMSs can manage some of your resources. Determine whether to configure an agency based on the actual service scenario.
+For example, you can configure an agency of the ECS type to automatically obtain the AK/SK to access OBS. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).
+The MRS_ECS_DEFAULT_AGENCY agency has the OBSOperateAccess permission of OBS and the CESFullAccess (for users who have enabled fine-grained policies), CES Administrator, and KMS Administrator permissions in the region where the cluster is located.
+Data Disk Encryption
+Whether to encrypt data in the data disk mounted to the cluster. This function is disabled by default. To use this function, you must have the Security Administrator and KMS Administrator permissions. This parameter is not available for MRS 3.1.0 and MRS 3.1.2-LTS.3.
+Keys used by encrypted data disks are provided by the Key Management Service (KMS) of the Data Encryption Workshop (DEW), secure and convenient. Therefore, you do not need to establish and maintain the key management infrastructure.
+Click Data Disk Encryption to enable or disable the data disk encryption function.
+Data Disk Key ID
+This parameter is displayed only when the Data Disk Encryption function is enabled. This parameter indicates the key ID corresponding to the selected key name. This parameter is not available for MRS 3.1.0 and MRS 3.1.2-LTS.3.
+Data Disk Key Name
+This parameter is mandatory when the Data Disk Encryption function is enabled. Select the name of the key used to encrypt the data disk. By default, the default master key named evs/default is selected. You can select another master key from the drop-down list. This parameter is not available for MRS 3.1.0 and MRS 3.1.2-LTS.3.
+If disks are encrypted using a CMK, which is then disabled or scheduled for deletion, the disks can no longer be read from or written to, and data on these disks may never be restored. Exercise caution when performing this operation.
+Click View Key List to enter a page where you can create and manage keys.
+Alarm
+If the alarm function is enabled, the cluster maintenance personnel can be notified in a timely manner to locate faults when the cluster runs abnormally or the system is faulty.
+Rule Name
+Name of the rule for sending alarm messages. The value can contain only digits, letters, hyphens (-), and underscores (_).
+Topic Name
+Select an existing topic or click Create Topic to create a topic. To deliver messages published to a topic, you need to add a subscriber to the topic. For details, see Adding Subscriptions to a Topic.
+A topic serves as a message sending channel, where publishers and subscribers can interact with each other.
+Logging
+Whether to collect logs when cluster creation fails.
+After the logging function is enabled, system logs and component run logs are automatically collected and saved to the OBS file system in scenarios such as cluster creation failures and scale-out or scale-in failures for O&M personnel to quickly locate faults. The log information is retained for a maximum of seven days.
Failed to Create a Cluster
If a cluster fails to be created, the failed task will be managed on the Manage Failed Tasks page. Choose Clusters > Active Clusters. Click
-
shown in Figure 1 to go to the Manage Failed Tasks page. In the Status column, hover the cursor over to view the failure cause. You can delete failed tasks by referring to Viewing Failed MRS Tasks.Table 5 lists the error codes of MRS cluster creation failures.
+Table 6 lists the error codes of MRS cluster creation failures.
-Table 5 Error codes Error Code
+Table 6 Error codes Error Code
Description
- Backup and Recovery Management
+
- SQL Inspector
+- Security Management
- Alarm Reference (Applicable to MRS 3.x)
diff --git a/docs/mrs/umn/mrs_01_0617.html b/docs/mrs/umn/mrs_01_0617.html index 40f00486..5602b4af 100644 --- a/docs/mrs/umn/mrs_01_0617.html +++ b/docs/mrs/umn/mrs_01_0617.html @@ -7,7 +7,7 @@diff --git a/docs/mrs/umn/mrs_01_0632.html b/docs/mrs/umn/mrs_01_0632.html index 4622cdd7..eebe8ad7 100644 --- a/docs/mrs/umn/mrs_01_0632.html +++ b/docs/mrs/umn/mrs_01_0632.html @@ -10,34 +10,36 @@-Parent topic: Using a Storage-Compute Decoupled Cluster+Parent topic: Interconnecting MRS with OBS Using an Agency-Prerequisites
- Fine-grained permission control has been enabled. For details about permissions management, see Creating an MRS User.
- You have a basic knowledge of IAM Agencies (see section "Agencies" in the IAM User Guide) and OBS fine-grained policies.
Step 1: Configuring an Agency with OBS Access Permission for a Cluster
-- Follow instructions in Configuring a Storage-Compute Decoupled Cluster (Agency) to configure an agency with OBS access permissions.
The agency takes effect for all users (including internal users) and user groups in the cluster. To control the permissions of users and user groups in the cluster to access OBS, perform the following operations.
-
Step 1: Configuring an Agency with OBS Access Permission for a Cluster
Follow instructions in Configuring a Storage-Compute Decoupled Cluster (Agency) to configure an agency with OBS access permissions.
+The agency takes effect for all users (including internal users) and user groups in the cluster. To control the permissions of users and user groups in the cluster to access OBS, perform the following operations.
+ When you configure permissions on an OBS path, if the write permission is configured, you need to configure the corresponding recycle bin path.
+The default recycle bin path is /user/${current.user}/.Trash/, in which ${current.user} indicates the current user.
+Step 2: Creating a Policy and an Agency on IAM
Create policies with different access permissions and bind the policies to the agency. For details, see Creating a Policy and an Agency on IAM.
Step 3: Configuring OBS Permission Control Mappings on the MRS Cluster Details Page
- On the MRS management console, choose Clusters > Active Clusters and click the cluster name.
- In the Basic Information area on the Dashboard tab page, click Manage next to OBS Permission Control.
- Click Add Mapping and set parameters according to Table 1.
-
Table 1 OBS permission control parameters Parameter
+Table 1 OBS permission control parameters Parameter
Description
+Description
IAM Agency
+IAM Agency
Select the agency created in 2.
+Select the agency created in 2.
Type
+Type
- User: User-level mapping
- Group: User group-level mapping
- User: User-level mapping
- Group: User group-level mapping
NOTE:- User-level mapping takes priority over user group-level mapping. If you select Group, you are advised to enter the primary group name in MRS User (User Group).
- Do not use the same username (user group) for multiple mapping records.
MRS User (User Group)
+MRS User (User Group)
Use commas (,) to separate multiple names of users or user groups.
+Use commas (,) to separate multiple names of users or user groups.
NOTE:@@ -64,36 +66,36 @@- If OBS permission control is not configured for a user and no AK and SK are configured, the OBS Operator permission in MRS_ECS_DEFAULT_AGENCY will be used for accessing OBS. You are advised not to bind the internal user of a component to an agency.
- If you need to configure an agency for the internal user of a component when submitting a job in the following scenarios, the requirements are as follows:
- To control permissions on spark-beeline operations, set the username to spark for a security cluster and omm for a normal cluster.
- To control permissions on HBase shell operations, set the username to hbase for a security cluster and omm for a normal cluster.
- To control permissions on Presto, set the username to omm, hive, and the username used for logging in to the client for a security cluster and omm and the username used for logging in to the client for a normal cluster.
- If you want to use Hive to create tables in beeline mode, set the username to the internal user hive.
Creating a Policy and an Agency on IAM
- Create a policy on IAM.
- Log in to the IAM console.
- Choose Permissions. On the displayed page, click Create Custom Policy.
- Set parameters according to Table 2. Obtain the customized OBS policy samples that are frequently used by referring to .
-
Table 2 Policy parameters Parameter
+- diff --git a/docs/mrs/umn/mrs_03_0002.html b/docs/mrs/umn/mrs_03_0002.html index 56c66270..44acec7d 100644 --- a/docs/mrs/umn/mrs_03_0002.html +++ b/docs/mrs/umn/mrs_03_0002.html @@ -6,8 +6,6 @@Table 2 Policy parameters Parameter
Description
+Description
Policy Name
+Policy Name
Only letters, digits, spaces, and special characters (-_.,) are allowed.
+Only letters, digits, spaces, and special characters (-_.,) are allowed.
Scope
+Scope
Select Global services, because OBS is a global service.
+Select Global services, because OBS is a global service.
Policy View
+Policy View
Select Visual editor.
+Select Visual editor.
Policy Content
+Policy Content
- Allow: Select Allow.
- Select service: Select Object Storage Service (OBS).
- Select action: Select WriteOnly, ReadOnly, and ListOnly.
- Specific resources:
- Set object to Specify resource path, click Add Resource Path, and enter obs_bucket_name/tmp/ and obs_bucket_name/tmp/*. The /tmp directory is used as an example. If you need to add permissions for other directories, perform the following steps to add the directories and resource paths of all objects in the directories.
- Set bucket to Specify resource path, click Add Resource Path, and enter obs_bucket_name.
- Allow: Select Allow.
- Select service: Select Object Storage Service (OBS).
- Select action: Select WriteOnly, ReadOnly, and ListOnly.
- Specific resources:
- Set object to Specify resource path, click Add Resource Path, and enter obs_bucket_name/tmp/ and obs_bucket_name/tmp/*. The /tmp directory is used as an example. If you need to add permissions for other directories, perform the following steps to add the directories and resource paths of all objects in the directories.
- Set bucket to Specify resource path, click Add Resource Path, and enter obs_bucket_name.
- (Optional) Add request condition, which does not need to be added currently.
Description
+Description
(Optional) Brief description about the policy.
+(Optional) Brief description about the policy.
diff --git a/docs/mrs/umn/mrs_01_0643.html b/docs/mrs/umn/mrs_01_0643.html index f89debcf..32d58b82 100644 --- a/docs/mrs/umn/mrs_01_0643.html +++ b/docs/mrs/umn/mrs_01_0643.html @@ -1,16 +1,13 @@ - --Parent topic: MRS Multi-User Permission Management+Using a Storage-Compute Decoupled Cluster
- -+Interconnecting MRS with OBS Using an Agency
+- diff --git a/docs/mrs/umn/mrs_01_0768.html b/docs/mrs/umn/mrs_01_0768.html index 2658d83f..069d5d05 100644 --- a/docs/mrs/umn/mrs_01_0768.html +++ b/docs/mrs/umn/mrs_01_0768.html @@ -12,10 +12,10 @@Step 2: Create a Cluster with Storage and Compute Separated
You can configure an agency when creating a cluster or bind an agency to an existing cluster to separate storage and compute. This section uses a cluster with Kerberos authentication enabled as an example.
Configuring an agency when creating a cluster:
-- Log in to the MRS management console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Custom Config tab.
- On the Custom Config tab page, set software parameters.
- Region: Select a region as required.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing.
- Cluster Version: Select a cluster version.
- Metadata: Select Local.
- Log in to the MRS management console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Custom Config tab.
- On the Custom Config tab page, set software parameters.
- Region: Select a region as required.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing.
- Cluster Version: Select a cluster version.
- Cluster Type: Select Analysis cluster or Hybrid cluster and select all components.
- Metadata: Select Local.
- Click Next and set hardware parameters.
- AZ: Use the default value.
- VPC: Use the default value.
- Subnet: Use the default value.
- Security Group: Use the default value.
- EIP: Use the default value.
- Cluster Node: Select the number of cluster nodes and node specifications based on site requirements.
- Click Next and set related parameters.
- Kerberos Authentication: This function is enabled by default. You can enable or disable it.
- Username: The default username is admin, which is used to log in to MRS Manager.
- Password: Set a password for user admin.
- Confirm Password: Enter the password of user admin again.
- Key Pair: Select a key pair from the drop-down list to log in to an ECS. Select "I acknowledge that I have obtained private key file SSHkey-xxx and that without this file I will not be able to log in to my ECS." If you have never created a key pair, click View Key Pair to create or import a key pair. And then, obtain a private key file.
- In this example, configure an agency and leave other parameters blank. For details about how to configure other parameters, see (Optional) Advanced Configuration.
Agency: Select the agency created in (Optional) Step 1: Create an ECS Agency with OBS Access Permissions or MRS_ECS_DEFAULT_AGENCY preset in IAM.
+ - In this example, configure an agency and leave other parameters blank. For details about how to configure other parameters, see Advanced Options.
Agency: Select the agency created in (Optional) Step 1: Create an ECS Agency with OBS Access Permissions or MRS_ECS_DEFAULT_AGENCY preset in IAM.
- To enable secure communications, select Enable. For details, see Communication Security Authorization.
- Click Apply Now and wait until the cluster is created.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
Configuring an agency for an existing cluster:
@@ -57,7 +57,6 @@ - Decompress the downloaded user credential file, and save the obtained krb5.conf and user.keytab files to the client directory, for example, /opt/Bigdata/client/Presto/.
- Run the following command to obtain a user principal:
- For clusters with Kerberos authentication enabled, run the following command to connect to the Presto Server of the cluster:
presto_cli.sh --krb5-config-path {krb5.conf file path} --krb5-principal {user principal} --krb5-keytab-path {user.keytab file path} --user {presto username}
- krb5.conf file path: Replace it with the file path set in 6.e, for example, /opt/Bigdata/client/Presto/krb5.conf.
- user.keytab file path: Replace it with the file path set in 6.e, for example, /opt/Bigdata/client/Presto/user.keytab.
- user principal: Replace it with the result returned in 6.f.
- presto username: Replace it with the name of the user created in 6.b, for example, presto001.
Example: presto_cli.sh --krb5-config-path /opt/Bigdata/client/Presto/krb5.conf --krb5-principal prest001@xxx_xxx_xxx_xxx.COM --krb5-keytab-path /opt/Bigdata/client/Presto/user.keytab --user presto001
- On the Presto client, run the following statement to create a schema and set location to an OBS path:
CREATE SCHEMA hive.demo01 WITH (location = 'obs://mrs-word001/presto-demo002/');
- Create a table in the schema. The table data is stored in the OBS file system. The following is an example.
CREATE TABLE hive.demo01.demo_table WITH (format = 'ORC') AS SELECT * FROM tpch.sf1.customer;
Figure 7 Return result@@ -72,7 +71,7 @@
diff --git a/docs/mrs/umn/mrs_01_1286.html b/docs/mrs/umn/mrs_01_1286.html index 4e390139..5be111af 100644 --- a/docs/mrs/umn/mrs_01_1286.html +++ b/docs/mrs/umn/mrs_01_1286.html @@ -34,7 +34,7 @@-Parent topic: Configuring a Cluster with Storage and Compute Decoupled+diff --git a/docs/mrs/umn/mrs_01_1288.html b/docs/mrs/umn/mrs_01_1288.html index 54e36b11..c8e4313a 100644 --- a/docs/mrs/umn/mrs_01_1288.html +++ b/docs/mrs/umn/mrs_01_1288.html @@ -11,7 +11,7 @@-Parent topic: Using a Storage-Compute Decoupled Cluster+Parent topic: Interconnecting MRS with OBS Using an Agencydiff --git a/docs/mrs/umn/mrs_01_1289.html b/docs/mrs/umn/mrs_01_1289.html index bd056dfb..4ce454ce 100644 --- a/docs/mrs/umn/mrs_01_1289.html +++ b/docs/mrs/umn/mrs_01_1289.html @@ -35,7 +35,7 @@-Parent topic: Using a Storage-Compute Decoupled Cluster+Parent topic: Interconnecting MRS with OBS Using an Agencydiff --git a/docs/mrs/umn/mrs_01_1292.html b/docs/mrs/umn/mrs_01_1292.html index 5ca44930..76a7357d 100644 --- a/docs/mrs/umn/mrs_01_1292.html +++ b/docs/mrs/umn/mrs_01_1292.html @@ -21,7 +21,7 @@-Parent topic: Using a Storage-Compute Decoupled Cluster+Parent topic: Interconnecting MRS with OBS Using an Agencydiff --git a/docs/mrs/umn/mrs_01_1298.html b/docs/mrs/umn/mrs_01_1298.html index 70c0d85c..ea4413b9 100644 --- a/docs/mrs/umn/mrs_01_1298.html +++ b/docs/mrs/umn/mrs_01_1298.html @@ -129,6 +129,8 @@-Parent topic: Using a Storage-Compute Decoupled Cluster+Parent topic: Interconnecting MRS with OBS Using an Agency- ALM-12089 Inter-Node Network Is Abnormal
+
- ALM-12091 Abnormal disaster Resources
+- ALM-12101 AZ Unhealthy
- ALM-12102 AZ HA Component Is Not Deployed Based on DR Requirements
@@ -143,6 +145,12 @@- ALM-12180 Suspended Disk I/O
+
- ALM-12186 CGroup Task Usage Exceeds the Threshold
+
+- ALM-12187 Failed to Expand Disk Partition Capacity
+
+- ALM-12188 diskmgt Disk Monitoring Unavailable
+- ALM-12190 Number of Knox Connections Exceeds the Threshold
- ALM-13000 ZooKeeper Service Unavailable
@@ -225,6 +233,16 @@- ALM-14030 HDFS Allows Write of Single-Replica Data
+
- ALM-14031 DataNode Process Is Abnormal
+
+- ALM-14032 JournalNode Process Is Abnormal
+
+- ALM-14033 ZKFC Process Is Abnormal
+
+- ALM-14034 Router Process Is Abnormal
+
+- ALM-14035 HttpFS Process Is Abnormal
+- ALM-16000 Percentage of Sessions Connected to the HiveServer to Maximum Number Allowed Exceeds the Threshold
- ALM-16001 Hive Warehouse Space Usage Exceeds the Threshold
@@ -339,6 +357,16 @@- ALM-19021 Handler Usage of RegionServer Exceeds the Threshold
+
- ALM-19022 HBase Hotspot Detection Is Unavailable
+
+- ALM-19023 Region Traffic Restriction for HBase
+
+- ALM-19024 RPC Requests P99 Latency on RegionServer Exceeds the Threshold
+
+- ALM-19025 Damaged StoreFile in HBase
+
+- ALM-19026 Damaged WAL Files in HBase
+- ALM-20002 Hue Service Unavailable
- ALM-23001 Loader Service Unavailable
@@ -391,6 +419,10 @@- ALM-25006 Sssd Service Exception
+
- ALM-25007 Number of SlapdServer Connections Exceeds the Threshold
+
+- ALM-25008 SlapdServer CPU Usage Exceeds the Threshold
+- ALM-25500 KrbServer Service Unavailable
- ALM-26051 Storm Service Unavailable
@@ -413,6 +445,24 @@- ALM-27007 Database Enters the Read-Only Mode
+
- ALM-29007 Impalad Process Memory Usage Exceeds the Threshold
+
+- ALM-29008 Number of ODBC Connections to Impalad Exceeds the Threshold
+
+- ALM-29010 Number of Queries Being Submitted by Impalad Exceeds the Threshold
+
+- ALM-29011 Number of Queries Being Executed by Impalad Exceeds the Threshold
+
+- ALM-29012 Number of Queries Being Waited by Impalad Exceeds the Threshold
+
+- ALM-29013 Impalad FGC Time Exceeds the Threshold
+
+- ALM-29014 Catalog FGC Time Exceeds the Threshold
+
+- ALM-29015 Catalog Process Memory Usage Exceeds the Threshold
+
+- ALM-29016 Impalad Instance in the Sub-healthy State
+- ALM-38000 Kafka Service Unavailable
- ALM-38001 Insufficient Kafka Disk Capacity
@@ -477,6 +527,8 @@- ALM-45001 Faulty HetuEngine Compute Instances
+
- ALM-45003 HetuEngine QAS Disk Capacity Is Insufficient
+- ALM-45175 Average Time for Calling OBS Metadata APIs Is Greater than the Threshold
- ALM-45176 Success Rate of Calling OBS Metadata APIs Is Lower than the Threshold
@@ -521,6 +573,14 @@- ALM-45288 TagSync Garbage Collection (GC) Time Exceeds the Threshold
+
- ALM-45289 PolicySync Heap Memory Usage Exceeds the Threshold
+
+- ALM-45290 PolicySync Direct Memory Usage Exceeds the Threshold
+
+- ALM-45291 PolicySync Non-Heap Memory Usage Exceeds the Threshold
+
+- ALM-45292 PolicySync GC Duration Exceeds the Threshold
+- ALM-45425 ClickHouse Service Unavailable
- ALM-45426 ClickHouse Service Quantity Quota Usage in ZooKeeper Exceeds the Threshold
@@ -541,6 +601,26 @@- ALM-45434 A Single Replica Exists in the ClickHouse Data Table
+
- ALM-45435 Inconsistent Metadata of ClickHouse Tables
+
+- ALM-45436 Skew ClickHouse Table Data
+
+- ALM-45437 Excessive Parts in the ClickHouse Table
+
+- ALM-45438 ClickHouse Disk Usage Exceeds 80%
+
+- ALM-45439 ClickHouse Node Enters the Read-Only Mode
+
+- ALM-45440 Inconsistency Between ClickHouse Replicas
+
+- ALM-45441 Zookeeper Disconnected
+
+- ALM-45442 Too Many Concurrent SQL Statements
+
+- ALM-45443 Slow SQL Queries in the Cluster
+
+- ALM-45444 Abnormal ClickHouse Process
+- ALM-45585 IoTDB Service Unavailable
- ALM-45586 IoTDBServer Heap Memory Usage Exceeds the Threshold
@@ -581,10 +661,106 @@- ALM-45638 Number of Restarts After Flink Job Failures Exceeds the Threshold
+
- ALM-45639 Checkpointing of a Flink Job Times Out
+- ALM-45640 FlinkServer Heartbeat Interruption Between the Active and Standby Nodes
- ALM-45641 Data Synchronization Exception Between the Active and Standby FlinkServer Nodes
+
- ALM-45642 RocksDB Continuously Triggers Write Traffic Limiting
+
+- ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold
+
+- ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold
+
+- ALM-45645 Pending Flush Size of RocksDB Continuously Exceeds the Threshold
+
+- ALM-45646 Pending Compaction Size of RocksDB Continuously Exceeds the Threshold
+
+- ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold
+
+- ALM-45648 RocksDB Frequently Encounters Write-Stopped
+
+- ALM-45649 P95 Latency of RocksDB Get Requests Continuously Exceeds the Threshold
+
+- ALM-45650 P95 Latency of RocksDB Write Requests Continuously Exceeds the Threshold
+
+- ALM-45652 Flink Service Unavailable
+
+- ALM-45653 Invalid Flink HA Certificate File
+
+- ALM-45654 Flink HA Certificate Is About to Expire
+
+- ALM-45655 Flink HA Certificate File Has Expired
+
+- ALM-45736 Guardian Service Unavailable
+
+- ALM-45737 TokenServer Heap Memory Usage Exceeds the Threshold
+
+- ALM-45738 TokenServer Direct Memory Usage Exceeds the Threshold
+
+- ALM-45739 TokenServer Non-Heap Memory Usage Exceeds the Threshold
+
+- ALM-45740 TokenServer GC Duration Exceeds the Threshold
+
+- ALM-45741 Failed to Call the ECS securitykey API
+
+- ALM-45742 Failed to Call the ECS Metadata API
+
+- ALM-45743 Failed to Call the IAM API
+
+- ALM-50201 Doris Service Unavailable
+
+- ALM-50202 FE CPU Usage Exceeds the Threshold
+
+- ALM-50203 FE Memory Usage Exceeds the Threshold
+
+- ALM-50205 BE CPU Usage Exceeds the Threshold
+
+- ALM-50206 BE Memory Usage Exceeds the Threshold
+
+- ALM-50207 Ratio of Connections to the FE MySQL Port to the Maximum Connections Allowed Exceeds the Threshold
+
+- ALM-50208 Failures to Clear Historical Metadata Image Files Exceed the Threshold
+
+- ALM-50209 Failures to Generate Metadata Image Files Exceed the Threshold
+
+- ALM-50210 Maximum Compaction Score of All BE Nodes Exceeds the Threshold
+
+- ALM-50211 FE Queue Length of BE Periodic Report Tasks Exceeds the Threshold
+
+- ALM-50212 Accumulated Old-Generation GC Duration of the FE Process Exceeds the Threshold
+
+- ALM-50213 Number of Tasks Queuing in the FE Thread Pool for Interacting with BE Exceeds the Threshold
+
+- ALM-50214 Number of Tasks Queuing in the FE Thread Pool for Task Processing Exceeds the Threshold
+
+- ALM-50215 Longest Duration of RPC Requests Received by Each FE Thrift Method Exceeds the Threshold
+
+- ALM-50216 Memory Usage of the FE Node Exceeds the Threshold
+
+- ALM-50217 Heap Memory Usage of the FE Node Exceeds the Threshold
+
+- ALM-50219 Length of the Queue in the Thread Pool for Query Execution Exceeds the Threshold
+
+- ALM-50220 Error Rate of TCP Packet Receiving Exceeds the Threshold
+
+- ALM-50221 BE Data Disk Usage Exceeds the Threshold
+
+- ALM-50222 Disk Status of a Specified Data Directory on BE Is Abnormal
+
+- ALM-50223 Maximum Memory Required by BE Is Greater Than the Remaining Memory of the Machine
+
+- ALM-50224 Failures a Certain Task Type on BE Are Increasing
+
+- ALM-50225 FE Instance Fault
+
+- ALM-50226 BE Instance Fault
+
+- ALM-50401 Number of JobServer Jobs Waiting to Be Executed Exceeds the Threshold
+
+- ALM-50402 JobGateway Service Unavailable
+diff --git a/docs/mrs/umn/mrs_01_2354.html b/docs/mrs/umn/mrs_01_2354.html index a127715a..07823785 100644 --- a/docs/mrs/umn/mrs_01_2354.html +++ b/docs/mrs/umn/mrs_01_2354.html @@ -2,7 +2,7 @@Quick Creation of a ClickHouse Cluster
This section describes how to quickly create a ClickHouse cluster. ClickHouse is a columnar database management system used for online analysis. It features the ultimate compression rate and fast query performance. It is widely used in Internet advertisement, app and web traffic analysis, telecom, finance, and IoT fields.
-Quick Creation of a ClickHouse Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Quick Config tab.
- Configure basic cluster information. For details about the parameters, see Creating a Custom Cluster.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, Example: mrs_20201121.
- Cluster Version: Select the latest version, which is the default value. (The components provided by a cluster vary according to the cluster version. Select a cluster version based on site requirements.)
- Component: Select ClickHouse cluster.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- Enterprise Project: Use the default value.
- Cluster Node: Select the number of cluster nodes and node specifications based on site requirements. For MRS 3.x or later, the memory of the master node must be greater than 64 GB.
- Kerberos Authentication: Select whether to enable Kerberos authentication.
- Username: The default value is root/admin. User root is used to remotely log in to ECSs, and user admin is used to access the cluster management page.
- Password: Set a password for user root/admin.
- Confirm Password: Enter the password of user root/admin again.
Quick Creation of a ClickHouse Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Quick Config tab.
- Configure basic cluster information. For details about the parameters, see Creating a Custom Cluster.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, Example: mrs_20201121.
- Cluster Type: Use the default value.
- Cluster Version: Select the latest version, which is the default value. (The components provided by a cluster vary according to the cluster version. Select a cluster version based on site requirements.)
- Component: Select ClickHouse cluster.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- Enterprise Project: Use the default value.
- CPU Architecture: Use the default value.
- Cluster Nodes: Select the number of cluster nodes and node specifications based on site requirements. For MRS 3.x or later, the memory of the master node must be greater than 64 GB.
- Kerberos Authentication: Select whether to enable Kerberos authentication.
- Username: The default value is root/admin. User root is used to remotely log in to ECSs, and user admin is used to access the cluster management page.
- Password: Set a password for user root/admin.
- Confirm Password: Enter the password of user root/admin again.
- Select Enable to enable secure communications. For details, see Communication Security Authorization.
- Click Apply Now.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
- Click Back to Cluster List to view the cluster status. Click Access Cluster to view cluster details.
For details about cluster status during creation, see the description of the status parameters in Table 1.
It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.
diff --git a/docs/mrs/umn/mrs_01_2355.html b/docs/mrs/umn/mrs_01_2355.html index 5a918e05..9c7c09dc 100644 --- a/docs/mrs/umn/mrs_01_2355.html +++ b/docs/mrs/umn/mrs_01_2355.html @@ -4,7 +4,7 @@This section describes how to quickly create a real-time analysis cluster. The real-time analysis cluster uses Hadoop, Kafka, Flink, and ClickHouse to collect, analyze, and query a large amount of data in real time.
The real-time analysis cluster consists of the following components:
- MRS 3.1.0: Hadoop 3.1.1, Kafka 2.11-2.4.0, Flink 1.12.0, ClickHouse 21.3.4.25, ZooKeeper 3.5.6, and Ranger 2.0.0.
Quick Creation of a Real-time Analysis Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Quick Config tab.
- Configure basic cluster information. For details about the parameters, see Creating a Custom Cluster.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, Example: mrs_20201130.
- Cluster Version: Select the latest version, which is the default value. (The components provided by a cluster vary according to the cluster version. Select a cluster version based on site requirements.)
- Component: Select Real-time Analysis Cluster.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- Enterprise Project: Use the default value.
- Cluster Node: Select the number of cluster nodes and node specifications based on site requirements. For MRS 3.x or later, the memory of the master node must be greater than 64 GB.
- Kerberos Authentication: Select whether to enable Kerberos authentication.
- Username: The default value is root/admin. User root is used to remotely log in to ECSs, and user admin is used to access the cluster management page.
- Password: Set a password for user root/admin.
- Confirm Password: Enter the password of user root/admin again.
Quick Creation of a Real-time Analysis Cluster
- Log in to the MRS console.
- Click Create Cluster. The page for creating a cluster is displayed.
- Click the Quick Config tab.
- Configure basic cluster information. For details about the parameters, see Creating a Custom Cluster.
- Region: Use the default value.
- Cluster Name: You can use the default name. However, you are advised to include a project name abbreviation or date for consolidated memory and easy distinguishing, Example: mrs_20201130.
- Cluster Type: Use the default value.
- Cluster Version: Select the latest version, which is the default value. (The components provided by a cluster vary according to the cluster version. Select a cluster version based on site requirements.)
- Component: Select Real-time Analysis Cluster.
- AZ: Use the default value.
- VPC: Use the default value. If there is no available VPC, click View VPC to access the VPC console and create a new VPC.
- Subnet: Use the default value.
- Enterprise Project: Use the default value.
- CPU Architecture: Use the default value.
- Cluster Nodes: Select the number of cluster nodes and node specifications based on site requirements. For MRS 3.x or later, the memory of the master node must be greater than 64 GB.
- Kerberos Authentication: Select whether to enable Kerberos authentication.
- Username: The default value is root/admin. User root is used to remotely log in to ECSs, and user admin is used to access the cluster management page.
- Password: Set a password for user root/admin.
- Confirm Password: Enter the password of user root/admin again.
- Select Enable to enable secure communications. For details, see Communication Security Authorization.
- Click Apply Now.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
- Click Back to Cluster List to view the cluster status. Click Access Cluster to view cluster details.
For details about cluster status during creation, see the description of the status parameters in Table 1.
It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.
diff --git a/docs/mrs/umn/mrs_01_24171.html b/docs/mrs/umn/mrs_01_24171.html index 652cdd0e..113ba8e9 100644 --- a/docs/mrs/umn/mrs_01_24171.html +++ b/docs/mrs/umn/mrs_01_24171.html @@ -38,7 +38,7 @@
diff --git a/docs/mrs/umn/mrs_01_24186.html b/docs/mrs/umn/mrs_01_24186.html index 0ceb9996..b421cd9d 100644 --- a/docs/mrs/umn/mrs_01_24186.html +++ b/docs/mrs/umn/mrs_01_24186.html @@ -69,7 +69,7 @@ client.sinks.kafka_sink.kafka.bootstrap.servers = Kafka domain name. This parameter is mandatory for a security cluster, for example, hadoop.xxx.com. +client.sinks.kafka_sink.kafka.kerberos.domain.name = Kafka domain name. This parameter is mandatory for a security cluster. client.sinks.kafka_sink.requiredAcks = 0 client.sources.static_log_source.channels = static_log_channel @@ -142,7 +142,7 @@ client.sinks.kafka_sink.kafka.bootstrap.servers = Kafka domain name. This parameter is mandatory for a security cluster, for example, hadoop.xxx.com. +client.sinks.kafka_sink.kafka.kerberos.domain.name = Kafka domain name. This parameter is mandatory for a security cluster. client.sinks.kafka_sink.requiredAcks = 0 client.sources.static_log_source.channels = static_log_channel diff --git a/docs/mrs/umn/mrs_01_24279.html b/docs/mrs/umn/mrs_01_24279.html new file mode 100644 index 00000000..e187f800 --- /dev/null +++ b/docs/mrs/umn/mrs_01_24279.html @@ -0,0 +1,52 @@ + + +-Parent topic: Using a Storage-Compute Decoupled Cluster+Parent topic: Interconnecting MRS with OBS Using an AgencyInterconnecting Flume with OBS
++This section applies to MRS 3.x or later.
+Before performing the following operations, ensure that you have configured a storage-compute decoupled cluster by referring to Configuring a Storage-Compute Decoupled Cluster (Agency) or Configuring a Storage-Compute Decoupled Cluster (AK/SK).
+- Configure an agency.
- Log in to the MRS console. In the navigation pane on the left, choose Clusters > Active Clusters.
- Click the name of a cluster to go to the cluster details page.
- On the Dashboard page, click Synchronize on the right of IAM User Sync to synchronize IAM users.
- Click Manage Agency on the right of Agency, select the target agency, and click OK.
- Create an OBS file system for storing data.
- Log in to the OBS console.
- In the navigation pane on the left, choose Parallel File Systems. On the displayed page, click Create Parallel File System.
- Enter the file system name, for example, esdk-c-test-pfs1, and set other parameters as required. Click Create Now.
- In the parallel file system list on the OBS console, click the created file system name to go to its details page.
- In the navigation pane on the left, choose Files and click Create Folder to create the testFlumeOutput folder.
- Prepare the properties.properties file and upload it to the /opt/flumeInput directory.
- Prepare the properties.properties file on the local host. Its content is as follows:
# source +server.sources = r1 +# channels +server.channels = c1 +# sink +server.sinks = obs_sink +# ----- define net source ----- +server.sources.r1.type = seq +server.sources.r1.spooldir = /opt/flumeInput +# ---- define OBS sink ---- +server.sinks.obs_sink.type = hdfs +server.sinks.obs_sink.hdfs.path = obs://esdk-c-test-pfs1/testFlumeOutput +server.sinks.obs_sink.hdfs.filePrefix = %[localhost] +server.sinks.obs_sink.hdfs.useLocalTimeStamp = true +# set file size to trigger roll +server.sinks.obs_sink.hdfs.rollSize = 0 +server.sinks.obs_sink.hdfs.rollCount = 0 +server.sinks.obs_sink.hdfs.rollInterval = 5 +#server.sinks.obs_sink.hdfs.threadsPoolSize = 30 +server.sinks.obs_sink.hdfs.fileType = DataStream +server.sinks.obs_sink.hdfs.writeFormat = Text +server.sinks.obs_sink.hdfs.fileCloseByEndEvent = false + +# define channel +server.channels.c1.type = memory +server.channels.c1.capacity = 1000 +# transaction size +server.channels.c1.transactionCapacity = 1000 +server.channels.c1.byteCapacity = 800000 +server.channels.c1.byteCapacityBufferPercentage = 20 +server.channels.c1.keep-alive = 60 +server.sources.r1.channels = c1 +server.sinks.obs_sink.channel = c1
+ + - Log in to the node where the Flume client is installed as user root.
- Create the /opt/flumeInput directory and create a customized .txt file in this directory.
- Log in to MRS Manager.
- Choose Cluster > Name of the target cluster > Services > Flume. On the displayed page, click Configurations and then Upload File in the Value column corresponding to the flume.config.file parameter, upload the properties.properties file prepared in 3.a, and click Save.
- Prepare the properties.properties file on the local host. Its content is as follows:
- View the result in the OBS system.
- Log in to the OBS console.
- Click Parallel File Systems and go to the folder created in 2 to view the result.
++ diff --git a/docs/mrs/umn/mrs_01_24294.html b/docs/mrs/umn/mrs_01_24294.html index 34955e52..270f7e46 100644 --- a/docs/mrs/umn/mrs_01_24294.html +++ b/docs/mrs/umn/mrs_01_24294.html @@ -209,7 +209,7 @@++Parent topic: Interconnecting MRS with OBS Using an Agency+
diff --git a/docs/mrs/umn/mrs_01_24297.html b/docs/mrs/umn/mrs_01_24297.html index e3e9379d..7b5b45c7 100644 --- a/docs/mrs/umn/mrs_01_24297.html +++ b/docs/mrs/umn/mrs_01_24297.html @@ -5,6 +5,8 @@-Parent topic: Using a Storage-Compute Decoupled Cluster+Parent topic: Interconnecting MRS with OBS Using an Agency-
+
- Quickly Creating a Doris Cluster
+ - Quick Creation of a Hadoop Analysis Cluster
- Quick Creation of an HBase Analysis Cluster
diff --git a/docs/mrs/umn/mrs_01_248978.html b/docs/mrs/umn/mrs_01_248978.html new file mode 100644 index 00000000..425821d8 --- /dev/null +++ b/docs/mrs/umn/mrs_01_248978.html @@ -0,0 +1,156 @@ + + +Interconnecting the Guardian Service with OBS
+++Scenario
This section describes how to enable storage and compute decoupling for the Guardian component. After this feature is enabled, Guardian can provide temporary authentication credentials for services such as HDFS, Hive, Spark, Loader, and HetuEngine to access OBS when decoupled storage and compute are used.
+Perform the following steps to interconnect Guardian with OBS:
+- Creating an OBS Parallel File System
- Creating a Cloud Service Agency and Binding It to a Cluster
- Creating an Agency for a Regular Account
- Configuring a Cloud Service Agency
- Granting OBS Access Permission to Guardian
- Enabling Cascading Authorization for Hive Tables
- Configuring the Recycle Bin Cleanup Policy
+Prerequisites
- Components such as Guardian, Ranger, and Hadoop have been installed in the cluster.
- If Guardian is installed after components such as Hadoop, HetuEngine, Hive, and Spark are installed, the Guardian client must be downloaded again and the default client for job submission on the management plane must be refreshed.
+Impact on the System
- After the configuration is complete, you need to refresh the configuration of the original client or reinstall the client.
- To submit a job on console, log in to the active OMS node as user omm and run the sh /opt/executor/bin/refresh-client-config.sh command to refresh the built-in client of the cluster.
+Creating an OBS Parallel File System
- Log in to the OBS console.
- Choose Parallel File Systems > Create Parallel File System.
- Enter a file system name, for example, guardian-obs.
The name of an enterprise project must be the same as that of the MRS cluster. Set other parameters.
+ - Click Create Now.
+Creating a Cloud Service Agency and Binding It to a Cluster
- Log in to the management console.
- In the service list, choose Management & Governance > Identity and Access Management.
- Choose Agencies. On the displayed page, click Create Agency.
- Set the agency name, for example, mrs_ecs_obs.
- Set Agency Type to Cloud service and select Elastic Cloud Server (ECS) and Bare Metal Server (BMS) to authorize ECS or BMS to invoke OBS.
- Set Validity Period to Unlimited and click Next.
- On the displayed page, search for the OBS OperateAccess policy and enable it.
- Click Next, select All resources, click Show More, select Global resources, and click OK.
- In the dialog box that is displayed, click OK to start authorization. After "Authorization successful." is displayed, click Finish. The agency is successfully created.
- Log in to the MRS console. In the navigation pane on the left, choose Active Clusters.
- Click the name of the target cluster to go to details page.
- In the Dashboard tab, click Synchronize on the right of IAM User Sync to synchronize the IAM user.
- In the Dashboard tab, click Manage Agency on the right of Agency, select the created agency, for example, mrs_ecs_obs, and click OK to bind the agency to the cluster.
+Creating an Agency for a Regular Account
- Log in to the management console.
- In the service list, choose Management & Governance > Identity and Access Management.
- Choose Agencies. On the displayed page, click Create Agency.
- Enter an agency name, for example, agency-MRS-to-OBS.
- Set Agency Type to Account.
- Enter your cloud account for Delegated Account, that is, the account you register using your mobile phone number. It cannot be a federated user or an IAM user created using your cloud account.
- Set Validity Period to Unlimited and click Next.
- On the displayed page, search for the OBS Administrator policy and enable it.
- Click Next, select All resources, click Show More, select Global resources, and click OK.
- After the agency is created, check and record the agency ID.
+Configuring a Cloud Service Agency
- Log in to the management console.
- In the service list, choose Management & Governance > Identity and Access Management.
- Select Agencies and click the agency mrs_ecs_obs created in Creating a Cloud Service Agency and Binding It to a Cluster
- Choose Permissions > Authorize, click Create Policy in the upper right corner, and set the parameters as follows:
- Policy Name: Enter a policy name, for example, guardian-assume-policy.
- Policy View: Select JSON.
- Policy Content: Configure the policy as follows. {Agency ID} indicates the ID recorded in 10.
{ + "Version": "1.1", + "Statement": [ + { + "Action": [ + "iam:agencies:assume" + ], + "Resource": { + "uri": [ + "/iam/agencies/{Agency ID}" + ] + }, + "Effect": "Allow" + } + ] +}+
- Click Next. On the Select Policy/Role page, select the policy created in 4.
- Click Next, select All resources, click Show More, select Global resources, and click OK.
+Granting OBS Access Permission to Guardian
- Log in to FusionInsight Manager, choose Cluster > Services > Guardian, click Configurations, and then All Configurations. On the displayed page, search for and modify the following parameters.
++
+Parameter
+Description
+Value
+fs.obs.guardian.accesslabel.enabled
+Whether to enable access label for using Guardian to connect to OBS.
+true
+fs.obs.guardian.enabled
+Whether to enable Guardian.
+true
+fs.obs.delegation.token.providers
+Delegation token generator. If fs.obs.guardian.enabled is set to true, configure both com.example.mrs.dt.MRSDelegationTokenProvider and com.example.mrs.dt.GuardianDTProvider.
+com.example.mrs.dt.MRSDelegationTokenProvider and com.example.mrs.dt.GuardianDTProvider
+token.server.access.label.agency.name
+Name of the specified IAM agency, which is the agency created in Creating an Agency for a Regular Account.
+agency-MRS-to-OBS
+ - Save the service configuration, choose More > Restart Configuration-Expired Instances on the FusionInsight Manager home page, and restart all service instances whose configurations have expired as prompted.
- To submit jobs on the MRS console, log in to the active OMS node as user omm and run the following command to refresh the built-in client configuration:
sh /opt/executor/bin/refresh-client-config.sh
+
+Enabling Cascading Authorization for Hive Tables
- Log in to FusionInsight Manager, choose Cluster > Services > Ranger and click Configurations.
- Search for the ranger.ext.authorization.cascade.enable parameter and set it to true.
+
- Click Save.
- Click Instance and select all RangerAdmin instances. Click More and select Restart Instance. Enter the password, and click OK to restart all RangerAdmin instances.
+Configuring the Recycle Bin Cleanup Policy
- Log in to the OBS Console.
- Select Parallel File Systems and click the file system created in Creating an OBS Parallel File System.
- Choose Basic Configurations > Lifecycle Rules and click Create to create a lifecycle rule for the /user/.Trash directory.+ +
For clusters that use decoupled storage and compute, configure a lifecycle policy for the related directories by referring to this chapter. Otherwise, the storage space may be used up and storage fees may increase.
++
+Table 1 Parameters for creating a lifecycle rule Parameter
+Description
+Example
+Status
+Whether to enable the lifecycle rule.
+Enable
+Rule Name
+User-defined rule name, which is used to identify different lifecycle configurations.
+rule-test
+Prefix
+Prefix of the objects to which the lifecycle rule applies. Generally, the recycle bin directory of MRS components is /user/.Trash.
+user/.Trash
+Transition to Infrequent Access After (Days)
+Number of days before transitioning to infrequent access after the object is last updated. The value of this parameter must be at least 30.
+30 days
+Transition to Archive After (Days)
+Number of days before transitioning to archive after the object is last updated. If Transition to Infrequent Access After (Days) is also configured, after the lifecycle is transitioned to infrequent access, wait at least 30 days before transitioning it to archive. If only Transition to Archive After (Days) is configured, there is no time limit.
+31 days
+Delete Files After (Days)
+Number of days before being deleted by OBS after the object is last updated. This parameter must be larger than the above two parameters.
+32 days
+Delete Fragments After (Days)
+Number of days before fragments are expired and deleted by OBS automatically.
+30 days
+ - Click OK to complete the lifecycle rule configuration.
To modify the lifecycle configuration, locate the lifecycle rule, click Edit or Disable on the right. Click Enable to enable the lifecycle rule.
+
++ diff --git a/docs/mrs/umn/mrs_01_248986.html b/docs/mrs/umn/mrs_01_248986.html new file mode 100644 index 00000000..5e1fb500 --- /dev/null +++ b/docs/mrs/umn/mrs_01_248986.html @@ -0,0 +1,20 @@ + + +++Parent topic: Interconnecting with OBS Using the Guardian Service+Interconnecting with OBS Using the Guardian Service
+++++ diff --git a/docs/mrs/umn/mrs_01_248987.html b/docs/mrs/umn/mrs_01_248987.html new file mode 100644 index 00000000..515171d2 --- /dev/null +++ b/docs/mrs/umn/mrs_01_248987.html @@ -0,0 +1,24 @@ + + +-
+
- Scenarios
+
+ - Interconnecting the Guardian Service with OBS
+
+ - Interconnecting Components with OBS Using Guardian
+
+
++Parent topic: Configuring a Cluster with Storage and Compute Decoupled+Scenarios
+++Configuring Storage and Compute Decoupling
- Create an MRS cluster.
The MRS cluster must contain basic components such as Guardian, Ranger, and Hadoop.
++ Currently, only MRS 3.3.0-LTS and later versions support interconnection with OBS using the Guardian component.
+ - Create an OBS agency.
Create an agency with OBS access permissions, which is used for interconnecting Guardian with OBS.
+ - Enable the interconnection between Guardian and OBS and configure parameters.
Modify the configuration parameters for the Guardian service and configure the IAM agency authentication information.
+ - Configure the policy for clearing component data in the recycle bin directory.
In the storage-compute decoupling scenario, the prevention against accidental deletion is enabled by default for components connected to OBS. When a user deletes data, the deleted object is moved to the corresponding recycle bin directory. You need to configure a lifecycle rule for the corresponding directory in the OBS file system to prevent the storage space from being used up.
+ - Interconnect components with OBS.
Components in the MRS cluster can directly access the corresponding path after the required permissions for accessing OBS buckets are obtained. You can use the component client to directly access resources in the OBS file system in absolute path mode.
+
+Configuring OBS Permissions
If Guardian is deployed with decoupled storage and compute and Ranger authentication is enabled for MRS clusters, Ranger administrators can configure read and write permissions on OBS directories or files for cluster users.
+With the Guardian permission model, storage and compute decoupling, and Hive cascading authorization, authorization is not required after the first permission service table authorization on the Ranger page and the system automatically associates the permissions of OBS data storage source in a fine-grained manner. The storage path of the table does not need to be sensed.
++ - On the Ranger page, OBS permission authorization only support Manager custom user groups (built-in user groups are not supported). The user group contains a maximum of 52 characters, including digits 0 to 9, letters A to Z, underscores (_), and number signs (#). Otherwise, the policy fails to be added.
- For clusters with Kerberos authentication enabled, permissions need to be granted based on Ranger. For clusters with Kerberos authentication disabled, OBS permissions are granted by default, and no additional configuration is required.
- If Kerberos authentication is not enabled for the current cluster, the user who accesses OBS must belong to the supergroup group.
++ diff --git a/docs/mrs/umn/mrs_01_248988.html b/docs/mrs/umn/mrs_01_248988.html new file mode 100644 index 00000000..fb4f273e --- /dev/null +++ b/docs/mrs/umn/mrs_01_248988.html @@ -0,0 +1,29 @@ + + +++Parent topic: Interconnecting with OBS Using the Guardian Service+Interconnecting Components with OBS Using Guardian
+ +++ diff --git a/docs/mrs/umn/mrs_01_248989.html b/docs/mrs/umn/mrs_01_248989.html new file mode 100644 index 00000000..47cfd50a --- /dev/null +++ b/docs/mrs/umn/mrs_01_248989.html @@ -0,0 +1,45 @@ + + +-
+
- Interconnecting Hive with OBS
+
+ - Interconnecting Flink with OBS
+
+ - Interconnecting Spark with OBS
+
+ - Interconnecting Hudi with OBS
+
+ - Interconnecting HetuEngine with OBS
+
+ - Interconnecting HDFS with OBS
+
+ - Interconnecting Yarn with OBS
+
+ - Interconnecting MapReduce with OBS
+
+
++Parent topic: Interconnecting with OBS Using the Guardian Service+Interconnecting Hive with OBS
+++Interconnecting with OBS
MRS clusters allow Hive to connect to OBS through Metastore.
+Interconnecting Hive with OBS through Metastore
+- You have configured storage and compute decoupling by referring to Interconnecting the Guardian Service with OBS.
- Log in to FusionInsight Manager and choose Cluster > Services > Hive, and click Configurations.
- Search for hive.metastore.warehouse.dir in the search box and change the parameter value to an OBS path, for example, obs://hivetest/user/hive/warehouse/. hivetest indicates the OBS file system name.Figure 1 hive.metastore.warehouse.dir configuration+
- Save the configuration, choose Cluster > Services, and restart the Hive service in the service list.
- Update the client configuration file.
- Log in to the node where the Hive client is located and run the following command to modify hivemetastore-site.xml in the Hive client configuration file directory:
vi Client installation directory/Hive/config/hivemetastore-site.xml
+ - Change the value of hive.metastore.warehouse.dir to the corresponding OBS path, for example, obs://hivetest/user/hive/warehouse/.
+
- Change the value of hive.metastore.warehouse.dir of hivemetastore-site.xml in the HCatalog client configuration file directory to the corresponding OBS path, for example, obs://hivetest/user/hive/warehouse/.
vi Client installation directory/Hive/HCatalog/conf/hivemetastore-site.xml
+
- Log in to the node where the Hive client is located and run the following command to modify hivemetastore-site.xml in the Hive client configuration file directory:
- Go to the Hive Beeline CLI, create a database, and ensure that the location is an OBS path.
cd Client installation directory
+kinit Component operation user
+beeline
+create database testdb1;
+show create database testdb1;
+
+
+Configuring Ranger Permissions
- Granting the read and write permissions on OBS paths to the hive user group
- Log in to the Ranger web UI as the Ranger administrator. On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area, and assign the Read and Write permissions on the OBS storage path to the hive user group. If this operation is successful, all users in the hive group can access the Hive data warehouse path.
For example, assign the Read and Write permissions on the obs://hivetest/user/hive/warehouse/ directory to the hive user group:
+
+
- Choose Settings > Roles, click Add New Role, and create a role whose Role Name is hive.
+
- Log in to the Ranger web UI as the Ranger administrator. On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area, and assign the Read and Write permissions on the OBS storage path to the hive user group. If this operation is successful, all users in the hive group can access the Hive data warehouse path.
- Granting the read and write permissions on OBS paths to a custom user group
- Log in to FusionInsight Manager and choose System > Permission > User Group. On the displayed page, click Create User Group.
- Create a user group without a role, for example, hiveobs1, and bind the user group to the corresponding user.
- Log in to the Ranger management page as the rangeradmin user.
- On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area.
- Grant the Read and Write permissions on the OBS storage path to the hiveobs1 user group. In this case, all users bound to the hiveobs1 user group can access the Hive data warehouse path.
+
- Creating a database,table, or partition in a custom location and granting read and write permissions on OBS paths
- Log in to the Ranger web UI as the Ranger administrator.
- On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area, and assign the Read and Write permissions on the OBS storage path to the user group of the corresponding user.
For example, assign the Read and Write permissions on the obs://obs-test/test/ directory to the hgroup1 user group, as shown in the following figure.
+
+
- On the home page, click the component plug-in name Hive in the HADOOP SQL area, and add a URL policy that grants the Read and Write permissions on the OBS path to the user group of the corresponding user.
For example, create the hive_url_policy URL policy for the hgroup1 user group and assign the Read and Write permissions on the obs://obs-test/test/ directory to the user group, as shown in the following figure.
+
+
- Log in to the beeline client and set Location to the OBS file system path when creating a table.
cd Client installation directory
+kinit Component operation user
+beeline
+For example, to create a table named test whose Location is obs://obs-test/test/Database name/Table name, run the following command:
+create external table test(name string) location "obs://obs-test/test/Database name/Table name";
+
+ - To authorize a view chart, you need to grant the view chart permission and the physical table path permission corresponding to the view chart.
- Cascading authorization can be performed only on databases and tables, and cannot be on partitions. If a partition path is not in the table path, you need to manually authorize the partition path.
- Cascading authorization for Deny Conditions in the Hive Ranger policy is not supported. That is, the Deny Conditions permission only restricts the table permission and cannot generate the permission of the HDFS/OBS storage source.
- The permission of the HDFS Ranger policy is prior to that of the HDFS/OBS storage source generated by cascading authorization. If the HDFS Ranger permission has been set for the HDFS storage source of the table, the cascading permission does not take effect.
- alter operations cannot be performed on tables whose storage source is OBS after cascading authorization. To perform the alter operation, you need to grant the Read and Write permissions of the parent directory of the OBS table path to the corresponding user group.
++ diff --git a/docs/mrs/umn/mrs_01_248991.html b/docs/mrs/umn/mrs_01_248991.html new file mode 100644 index 00000000..d31ebad2 --- /dev/null +++ b/docs/mrs/umn/mrs_01_248991.html @@ -0,0 +1,36 @@ + + +++Parent topic: Interconnecting Components with OBS Using Guardian+Interconnecting Flink with OBS
+++Interconnecting with OBS
- Log in to the Flink client installation node as the client installation user.
- Run the following command to initialize environment variables:
source Client installation directory/bigdata_env
+ - Configure the Flink client.
- Start a session.
- Normal cluster (Kerberos authentication disabled) +
- Security cluster (Kerberos authentication enabled)
- If the flink.keystore and flink.truststore file paths are relative paths:
Run the following command in the directory at the same level as ssl to start the session. ssl/ is a relative path.
+cd /opt/hadoopclient/Flink/flink/conf/
+yarn-session.sh -t ssl/ -nm "session-name" -d
+... +Cluster started: Yarn cluster with application id application_1624937999496_0017 +JobManager Web Interface: http://192.168.1.150:32261
+ - If the flink.keystore and flink.truststore file paths are absolute paths:
Run the following command to start a session:
+cd /opt/hadoopclient/Flink/flink/conf/
+yarn-session.sh -nm "session-name" -d
+
- If the flink.keystore and flink.truststore file paths are relative paths:
- For a security cluster, run the following command to perform user authentication. If Kerberos authentication is not enabled for the current cluster, you do not need to run this command.
kinit Username
+ - Explicitly add the OBS file system to be accessed in the Flink command line.
echo -e 'test' >/tmp/test
+hdfs dfs -mkdir -p obs://Parallel file system name/tmp/flinkjob
+hdfs dfs -put /tmp/test/ obs://Parallel file system name/tmp/flinkjob/
+flink run Client installation directory/Flink/flink/examples/batch/WordCount.jar -input obs://Parallel file system name/tmp/flinkjob/test -output obs://Parallel file system name/tmp/flinkjob/output
+
+ - Flink jobs are running on Yarn. Before configuring Flink to interconnect with the OBS file system, ensure that the interconnection between Yarn and the OBS file system is normal.
- Name of the OBS parallel file system/File name: The OBS file path must be written to the directory level.
- If Kerberos authentication has been enabled (security mode) for the cluster, grant the Read and Write permissions on OBS paths to component users in Ranger by referring to Ranger Permission Configuration.
+Ranger Permission Configuration
- Log in to FusionInsight Manager and choose System > Permission > User Group. On the displayed page, click Create User Group.
- Create a user group without a role, for example, obs_flink, and bind the user group to the corresponding user.
- Log in to the Ranger management page as the rangeradmin user.
- On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area.
- Click Add New Policy to add the Read and Write permissions on OBS paths to the user group created in 2. If there are no OBS paths, create one in advance (wildcard character * is not allowed).
+
++ diff --git a/docs/mrs/umn/mrs_01_248992.html b/docs/mrs/umn/mrs_01_248992.html new file mode 100644 index 00000000..9869c8f3 --- /dev/null +++ b/docs/mrs/umn/mrs_01_248992.html @@ -0,0 +1,55 @@ + + +++Parent topic: Interconnecting Components with OBS Using Guardian+Interconnecting Spark with OBS
+++Interconnecting with OBS
In an MRS cluster, Location can be set to an OBS file system path during Spark table creation and Spark can connect to OBS through Hive Metastore.
+- Setting the location to an OBS path during table creation:
- Log in to the node where the client is installed as the client installation user and access the spark-sql client.
cd Client installation directory
+kinit Component operation user
+spark-sql --master yarn
+ - Set Location to the OBS file system path when creating a table.
For example, to create a table named test whose Location is obs://obs-test/test/Database name/Table name, run the following command:
+create external table testspark(name string) location "obs://obs-test/test/Database name/Table name";
+
- Log in to the node where the client is installed as the client installation user and access the spark-sql client.
- Interconnecting Spark with OBS through Hive Metastore:
- Complete the configurations by referring to Interconnecting the Guardian Service with OBS.
- Log in to FusionInsight Manager, choose Cluster > Services > Spark and choose Configurations > All Configurations.
- In the navigation pane on the left, choose SparkResource > Customization. In the custom configuration items, add spark.sql.warehouse.location.first to the custom parameter and set its value to true.Figure 1 spark.sql.warehouse.location.first configuration+
- In the navigation pane on the left, choose JDBCServer > Customization. In the custom configuration items, add spark.sql.warehouse.location.first to the custom parameter and set its value to true.Figure 2 spark.sql.warehouse.location.first configuration+
- Click Save to save the configuration. Click the Dashboard tab choose More > Restart Service, enter the password, click OK, and click OK again to restart Spark.
- After Spark is restarted, choose More > Download Client to download and install the Spark client again. Then, go to 7.If you do not download and install the client again, you can perform the following steps to update the Spark client configuration file (for example, the client directory is /opt/client):+
- Log in to the node where the Spark client is deployed as user root and switch to the client installation directory.
cd /opt/client
+ - Run the following command to modify hive-site.xml in the configuration file directory of the Spark client:
vi Spark/spark/conf/hive-site.xml
+Change the value of hive.metastore.warehouse.dir to the corresponding OBS path, for example, obs://hivetest/user/hive/warehouse/.
+<property> +<name>hive.metastore.warehouse.dir</name> +<value>obs://hivetest/user/hive/warehouse/<value> +</property>
+ - Run the following command to modify the spark-defaults.conf file in the configuration file directory of the Spark client and set spark.sql.warehouse.location.first to true:
vi Spark/spark/conf/spark-defaults.conf
+
- Log in to the node where the Spark client is deployed as user root and switch to the client installation directory.
- Configure the OBS directory permission for the component operation user in clusters with Kerberos authentication enabled by referring to Configuring Ranger Permissions.
- Go to the SparkSQL CLI and spark-beeline, create a table, and check whether the location of the table is an OBS path.
+
kinit Service user (skip this step for normal clusters)
+- Go to the SparkSQL CLI.
+
create table d(a int);
+desc formatted d;
+As shown in the following figure, the location of table d is in the specified OBS path.
+
+
- Go to spark-beeline.
+
create table e(a int);
+desc formatted e;
+As shown in the following figure, the location of table e is in the specified OBS path.
+
+
- Go to the SparkSQL CLI.
+
+Configuring Ranger Permissions
- Log in to FusionInsight Manager and choose System > Permission > User Group. On the displayed page, click Create User Group.
- Create a user group without a role, for example, obs_spark, and bind the user group to the corresponding user.
- Log in to the Ranger management page as the rangeradmin user.
- On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area.
- Click Add New Policy to add the Read and Write permissions on OBS paths to the user group created in 2. If there are no OBS paths, create one in advance (wildcard character * is not allowed).
+
+ - Cascading authorization is not supported for view tables.
- Cascading authorization can be performed only on databases and tables, and cannot be on partitions. If a partition path is not in the table path, you need to manually authorize the partition path.
- Cascading authorization for Deny Conditions in the Hive Ranger policy is not supported. That is, the Deny Conditions permission only restricts the table permission and cannot generate the permission of the HDFS storage source.
- The permission of the HDFS Ranger policy is prior to that of the HDFS storage source generated by cascading authorization. If the HDFS Ranger permission has been set for the HDFS storage source of the table, the cascading permission does not take effect.
+Configuring Permissions for CDL Service Users
If Kerberos authentication is enabled for the cluster (the cluster is in security mode) and you need to store real-time data to OBS after the interconnection, perform the following operations to grant the Read and Write permissions on the corresponding OBS path to the specific user:
+- Log in to FusionInsight Manager and choose System > Permission > User Group. On the displayed page, click Create User Group.
- Create a user group without a role, for example, obs_cdl, and bind the user group to the corresponding CDL service user, for example, cdluser.
- Log in to the Ranger management page as the rangeradmin user.
- On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area.
- Click Add New Policy to add the Read and Write permissions on OBS paths to the created user group. If there are no OBS paths, create one in advance (wildcard character * is not allowed).
The following figure shows the configurations needed for adding the Read and Write permissions on obs://OBS parallel file system name/cdldata to user group obs_cdl.
+
+
++ diff --git a/docs/mrs/umn/mrs_01_248993.html b/docs/mrs/umn/mrs_01_248993.html new file mode 100644 index 00000000..eb6ae874 --- /dev/null +++ b/docs/mrs/umn/mrs_01_248993.html @@ -0,0 +1,51 @@ + + +++Parent topic: Interconnecting Components with OBS Using Guardian+Interconnecting Hudi with OBS
+++Interconnecting with OBS
- Log in to the node where the client is installed as the client installation user.
- Run the following commands to configure environment variables:
sourceClient installation directory/bigdata_env
+source Client installation directory/Hudi/component_env
+ - Modify the configuration file.
vim Client installation directory/Hudi/hudi/conf/hdfs-site.xml
+<property> +<name>dfs.namenode.acls.enabled</name> +<value>false</value> +</property>
+ - If Kerberos authentication has been enabled(security mode) for the cluster, run the following command to perform authentication as a user who has the Read and Write permissions on the corresponding OBS path. If Kerberos authentication has not been enabled (normal mode) for the cluster, you do not need to run this command.
kinit Username
+ - Start spark-shell and run the following commands to create a COW table and store it to OBS:
spark-shell --master yarn
+import org.apache.hudi.QuickstartUtils._
+import scala.collection.JavaConversions._
+import org.apache.spark.sql.SaveMode._
+import org.apache.hudi.DataSourceReadOptions._
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.config.HoodieWriteConfig._
+val tableName = "hudi_cow_table"
+val basePath = "obs://testhudi/cow_table/"
+val dataGen = new DataGenerator
+val inserts = convertToStringList(dataGen.generateInserts(10))
+val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
+df.write.format("org.apache.hudi").
+options(getQuickstartWriteConfigs).
+option(PRECOMBINE_FIELD_OPT_KEY, "ts").
+option(RECORDKEY_FIELD_OPT_KEY, "uuid").
+option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
+option(TABLE_NAME, tableName).
+mode(Overwrite).
+save(basePath);
++ "obs://testhudi/cow_table/" is the OBS path, and testhudi is the name of the OBS parallel system file. Change them based on site requirements.
+ - Use DataSource to check whether the table is created and whether the data is normal.
val roViewDF = spark.
+read.
+format("org.apache.hudi").
+load(basePath + "/*/*/*/*")
+roViewDF.createOrReplaceTempView("hudi_ro_table")
+spark.sql("select * from hudi_ro_table").show()
+ - Run the :q command to exit the spark-shell CLI.
+Configuring Ranger Permissions
- Log in to FusionInsight Manager and choose System > Permission > User Group. On the displayed page, click Create User Group.
- Create a user group without a role, for example, obs_hudi, and bind the user group to the corresponding user.
- Log in to the Ranger management page as the rangeradmin user.
- On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area.
- Click Add New Policy to add the Read and Write permissions on OBS paths to the user group created in 2. If there are no OBS paths, create one in advance (wildcard character * is not allowed).
+
++ diff --git a/docs/mrs/umn/mrs_01_248995.html b/docs/mrs/umn/mrs_01_248995.html new file mode 100644 index 00000000..13540617 --- /dev/null +++ b/docs/mrs/umn/mrs_01_248995.html @@ -0,0 +1,27 @@ + + +++Parent topic: Interconnecting Components with OBS Using Guardian+Interconnecting HetuEngine with OBS
+++ +Interconnecting with OBS
In an MRS cluster, Location can be set to an OBS file system path during HetuEngine table creation and HetuEngine can connect to OBS through Hive Metastore.
+- Setting Location to the OBS file system path when creating a table
- If a HetuEngine compute instance is running, restart it.
Log in to FusionInsight Manager as a user who has permission to access the HetuEngine web UI. Choose Cluster > Services > HetuEngine. In the Basic Information area in the Dashboard tab, click the link next to HSConsole WebUI. On the displayed HSConsole page, click Compute Instance. In the instance list, click Restart in the Operation column and operate as prompted.
+ - Log in to the node where the HetuEngine service client is located as the client installation user and run the following command:
source Client installation directory/bigdata_env
+ - Log in to the HetuEngine client based on the cluster authentication mode.
- Kerberos authentication has been enabled for the cluster (security mode): Run the following command to complete user authentication and log in to the HetuEngine client:
kinit User performing HetuEngine operations
+hetu-cli --catalog hive --tenant default --schema default
+For details about how to assign permissions to users in the Ranger, see Configuring Ranger Permissions.
+ - Kerberos authentication is not enabled for the cluster (normal mode): Run the following command to log in to the HetuEngine client:
hetu-cli --catalog hive --tenant default --schema default --user User performing HetuEngine operations
+
- Kerberos authentication has been enabled for the cluster (security mode): Run the following command to complete user authentication and log in to the HetuEngine client:
- Set Location to the OBS file system path when creating a table.
create table test(name string) with (location = 'obs://Name of the OBS parallel file system/user/hive/warehouse/test');
+
- If a HetuEngine compute instance is running, restart it.
- Interconnecting with OBS through Hive Metastore
- Complete the configurations by referring to Interconnecting Hive with OBS using MetaStore.
- Log in to FusionInsight Manager, choose Cluster > Services > HetuEngine. On the displayed page, choose More > Synchronize Configuration. After the synchronization is complete, choose More > Synchronize Configuration again and then restart the HetuEngine service as prompted.+
If a HetuEngine compute instance is running, stop it before restarting the service. After the service is restarted, start this compute instance.
+ - No location needs to be specified when you log in to the HetuEngine client to create a schema or table. The schema or table is stored on OBS by default.
++ diff --git a/docs/mrs/umn/mrs_01_248996.html b/docs/mrs/umn/mrs_01_248996.html new file mode 100644 index 00000000..6a7a8dec --- /dev/null +++ b/docs/mrs/umn/mrs_01_248996.html @@ -0,0 +1,32 @@ + + +++Parent topic: Interconnecting Components with OBS Using Guardian+Interconnecting HDFS with OBS
+++Interconnecting with OBS
- Log in to the node on which the HDFS client is installed as a client installation user.
- Run the following command to switch to the client installation directory:
cd Client installation directory
+ - Run the following command to configure environment variables:
source bigdata_env
+ - If the cluster is in security mode, run the following command to authenticate the user. The user must have the read and write permissions on the OBS directory. Skip user authentication for normal clusters.
kinit User performing HDFS operations
+ - Explicitly add the OBS file system to be accessed in the HDFS command line.
The following commands are examples.
+- Access the OBS file system. +
- Create a directory in the OBS file system. +
- Upload the /opt/test.txt file on the client node to the obs://OBS parallel file system name/hadoop directory.
hdfs dfs -put /opt/test.txt obs://OBS parallel file system name/hadoop
+
+ If a large number of logs are printed in the OBS file system, the read and write performance may be affected. You can adjust the log level of the OBS client as follows:
+cd Client installation directory/HDFS/hadoop/etc/hadoop
+vi log4j.properties
+Add the OBS log level configuration to the file as follows:
+log4j.logger.org.apache.hadoop.fs.obs=WARN
+log4j.logger.com.obs=WARN
+
+
+Configuring Ranger Permissions
- Log in to FusionInsight Manager and choose System > Permission > User Group. On the displayed page, click Create User Group to create a user group without any roles, for example, obs_hadoop.
- Back to FusionInsight Manager and choose System > Permission > User. On the displayed page, click Create User to create a user that is associated only with the obs_hadoop user group, for example, hadoopuser.
- Log in to the Ranger management page as the rangeradmin user.
- On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area.
- Click Add New Policy and add the Read and Write permissions on the desired OBS paths to the created user group.
The following figure shows the configurations needed for adding the Read and Write permissions on obs://OBS parallel file system name/hadoop to user group obs_hadoop.
+
+
++ diff --git a/docs/mrs/umn/mrs_01_248997.html b/docs/mrs/umn/mrs_01_248997.html new file mode 100644 index 00000000..12099629 --- /dev/null +++ b/docs/mrs/umn/mrs_01_248997.html @@ -0,0 +1,37 @@ + + +++Parent topic: Interconnecting Components with OBS Using Guardian+Interconnecting Yarn with OBS
+++Interconnecting with OBS
- Log in to the node on which the Yarn client is installed as a client installation user.
- Run the following command to switch to the client installation directory.
cd Client installation directory
+ - Run the following command to configure environment variables:
source bigdata_env
+ - If the cluster is in security mode, run the following command to authenticate the user. The user must have the read and write permissions on the OBS directory. Skip user authentication for normal clusters.
kinit User performing HDFS operations
+ - Explicitly add the OBS file system to be accessed in the Yarn command line.
- Access the OBS file system. +
- Create a directory in the OBS file system. +
- Execute the Yarn task to access OBS.
yarn jar Client installation directory/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi -Dmapreduce.job.hdfs-servers=NAMESERVICE -fs obs://OBS parallel file system name 1 1
+
NAMESERVICE indicates the NameService in HDFS. The default value is hdfs://hacluster. If there are multiple NameServices, separate them with ,.
+The following command is an example:
+yarn jar /opt/hadoopclient/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi -Dmapreduce.job.hdfs-servers=hdfs://hacluster -fs obs://bucketname 1 1
+- Run the following command to write data to OBS:
yarn jar Client installation directory/HDFS/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar teragen 100 obs://OBS parallel file system name/hadoop1/teragen1
+ - Run the following command to copy data from OBS to HDFS:
hadoop distcp obs://OBS parallel file system name/hadoop1/teragen1 /tmp
+
+ If a large number of logs are printed in the OBS file system, the read and write performance may be affected. You can adjust the log level of the OBS client as follows:
+cd Client installation directory/Yarn/config
+vi log4j.properties
+Add the OBS log level configuration to the file. (If an application uses the built-in log4j.properties file, add the same configuration.)
+log4j.logger.org.apache.hadoop.fs.obs=WARN
+log4j.logger.com.obs=WARN
+
+
+Configuring Ranger Permissions
- Log in to FusionInsight Manager and choose System > Permission > User Group. On the displayed page, click Create User Group to create a user group without any roles, for example, obs_hadoop1.
- Back to FusionInsight Manager and choose System > Permission > User. On the displayed page, click Create User to create a user that is associated with the obs_hadoop1 user group and the default role, for example, hadoopuser1.
- Log in to the Ranger management page as the rangeradmin user.
- On the home page, click component plug-in name OBS in the EXTERNAL AUTHORIZATION area.
- Click Add New Policy and add the Read and Write permissions on the desired OBS paths to the user group created in 1.
The following figure shows the configurations needed for adding the Read and Write permissions on obs://OBS parallel file system name/hadoop1 to user group obs_hadoop1.
+
+
++ diff --git a/docs/mrs/umn/mrs_01_248998.html b/docs/mrs/umn/mrs_01_248998.html new file mode 100644 index 00000000..3645c037 --- /dev/null +++ b/docs/mrs/umn/mrs_01_248998.html @@ -0,0 +1,13 @@ + + +++Parent topic: Interconnecting Components with OBS Using Guardian+Interconnecting MapReduce with OBS
+++Interconnecting with OBS
- Log in to FusionInsight Manager, choose Cluster > Services > MapReduce and choose Configurations > All Configurations. In the navigation tree, choose MapReduce > Customization. In the customized configuration items, add the configuration item mapreduce.jobhistory.always-scan-user-dir to core-site.xml and set the parameter to true.
+
- Save the configurations and restart the MapReduce service.
++ diff --git a/docs/mrs/umn/mrs_01_249150.html b/docs/mrs/umn/mrs_01_249150.html new file mode 100644 index 00000000..7cde599a --- /dev/null +++ b/docs/mrs/umn/mrs_01_249150.html @@ -0,0 +1,125 @@ + + +++Parent topic: Interconnecting Components with OBS Using Guardian+Configuring the Policy for Clearing Component Data in the Recycle Bin
+++Scenario
By default, components in an MRS 3.2.0-LTS.1 or later cluster support prevention against accidental data deletion. Native HDFS garbage collection can be used in the Hadoop big data systems that use OBS.
+The file data deleted by a component user is not directly deleted, but is stored in the recycle bin of the OBS file system instead. This section describes how to set a lifecycle rule for the recycle bin directory to periodically clear related data.
++ - For clusters that use decoupled storage and compute, configure a lifecycle policy for the related directories by referring to this chapter. Otherwise, the storage space may be used up and storage fees may increase.
- The recycle bin directory is created per user. When a user is created in the MRS cluster and the user has the permission to delete component data, you need to configure the recycle bin clearing rule for this new user.
- For HBase components that use decoupled storage and compute in MRS 3.1.2 or later versions, refer to this topic to set a policy for clearing component data in the recycle bin.
You need to configure lifecycle policies for the recycle bin directories of preset users in the MRS cluster and the recycle bin directories of new users who need accidental deletion prevention. If a low privileged agency is used or only the permission for MRS users to access OBS file system directories is configured by referring to Configuring Fine-Grained Permissions for MRS Multi-User Access to OBS, you will need the operation permission for the recycle bin directory.
+ ++
+Table 1 Directories for which a lifecycle policy needs to be configured Cluster Version
+Directory Type
+Component
+Directory
+How to Create
+Versions earlier than MRS 3.3.0-LTS
+Recycle bin directories that must be configured by default for each component in an MRS cluster
+Hive
+- user/omm/.Trash
- user/hive/.Trash
If the .Trash folder does not exist, create it on the cluster client as user omm.
+Run the following command:
+hdfs dfs -mkdir -p obs://Name of the OBS parallel file system where the table is stored/Folder path
+Spark
+- user/omm/.Trash
- user/root/.Trash
- user/spark2x/.Trash
HetuEngine
+- user/omm/.Trash
- user/hetuserver/.Trash
HBase
+- user/hbase/.Trash
- user/omm/.Trash
Recycle bin directories of users who need accidental deletion prevention
+Hive/Spark/HetuEngine
+user/<New service user>/.Trash
+MRS 3.3.0-LTS or later
+Default recycle bin directories configured for each component in an MRS cluster
+Hive/Spark/HetuEngine
+/user/.Trash
+For example, if a new user in the cluster has the following permissions, you need to create a recycle bin directory clearing rule for the user in the parallel file system:
+- Permissions to delete the HDFS files
- DROP, INSERT OVERWRITE, and TRUNCATE permissions on Hive tables
- DROP, TRUNCATE, DELETE, INSERT OVERWRITE, and LOAD OVERWRITE permissions on HetuEngine
+Configuring the Lifecycle Rule of an OBS Directory
- Log in to the OBS console.
- Click Parallel File Systems and click the name of the file system used by the current MRS cluster.
- In the navigation pane on the left, choose Basic Configurations > Lifecycle Rules. Click Create to create a lifecycle rule for a specified directory..
+
+
+Table 2 Parameters for creating a lifecycle rule Name
+Description
+Example Value
+Status
+Whether to enable the lifecycle rule.
+Enable
+Rule Name
+Rule name that identifies different lifecycle configurations.
+rule-test
+Prefix
+Prefix of the objects to which the lifecycle rule applies. Objects that have the specified prefix will be managed by the lifecycle rule. The prefix cannot start with a slash (/), have consecutive slashes (/), or contain the following special characters: \:*?"<>| If this parameter is not specified, the rule will take effect for the entire file system.
+NOTE:+To prevent other service data from being deleted by mistake, you are not advised to use the lifecycle rule configured for the entire file system or high-level directories.
+Generally, the recycle bin directory of MRS components is in the following format. If the folder does not exist, create it.
+user/<Username>/.Trash
+user/omm/.Trash
+Delete Files After (Days)
+The object within the rule configuration scope expires and is automatically deleted by OBS if the number of days since its last update reaches this parameter value.
+30 days
+ - Click OK to complete the lifecycle rule configuration.
You can click Edit in the Operation column of a lifecycle rule to edit it. You can also click Disable or Enable to disable or enable it.
+ - Repeat the preceding steps to create recycle bin directory clearing rules for all users who have the data deletion permission in the current MRS cluster one by one until all recycle bin directories in the OBS file system are configured.
++ diff --git a/docs/mrs/umn/mrs_01_249279.html b/docs/mrs/umn/mrs_01_249279.html new file mode 100644 index 00000000..fcf0b371 --- /dev/null +++ b/docs/mrs/umn/mrs_01_249279.html @@ -0,0 +1,27 @@ + + + ++ ++Interconnecting with OBS Using the Cluster Agency Mechanism
+ +++ +++ diff --git a/docs/mrs/umn/mrs_01_249283.html b/docs/mrs/umn/mrs_01_249283.html new file mode 100644 index 00000000..0c75e72b --- /dev/null +++ b/docs/mrs/umn/mrs_01_249283.html @@ -0,0 +1,19 @@ + + +-
+
- Configuring a Storage-Compute Decoupled Cluster (Agency)
+
+ - Configuring a Storage-Compute Decoupled Cluster (AK/SK)
+
+ - Configuring the Policy for Clearing Component Data in the Recycle Bin
+
+ - Interconnecting MRS with OBS Using an Agency
+
+ - Configuring Fine-Grained Permissions for MRS Multi-User Access to OBS
+
+
++Parent topic: Configuring a Cluster with Storage and Compute Decoupled+Quickly Creating a Doris Cluster
++This section describes how to quickly create a Doris cluster. Doris is a high-performance, real-time analytical database, suitable for report analysis, ad hoc query, and federated data lake query acceleration.
+A Doris cluster contains only the Doris 1.2.3 component.
++Quickly Creating a Doris Cluster
- Log in to the MRS management console.
- Click Create Cluster. The page for creating a cluster is displayed.
- On the displayed page, click the Quick Config tab.
- Configure basic information as follows. For details about the parameters, see Creating a Custom Cluster.
- Region: Retain the default value.
- Cluster Name: You can use the default name or add the project name initials or dates for distinction, for example, mrs_20180321.
- Cluster Type: Select Doris Cluster.
- Cluster Version: The latest version is selected by default. (Components vary depending on the version type. Select a version type as needed.)
- Component: Select Doris Cluster.
- AZ: Retain the default value.
- Enterprise Project: Select the default project.
- VPC: Retain the default value. If there is no available VPC, click View VPC to access the VPC console and create one.
- Subnet: Retain the default value.
- CPU Architecture: Use the default value.
- Cluster Nodes: Set the number of cluster nodes and node specifications based on your needs. For MRS 3.x and later, the memory of a master node must be at least 64 GB.
- Kerberos Authentication: Whether to enable Kerberos authentication. This option cannot be changed after you create a cluster.
- Username: The default value is root/admin. User root is used to remotely log in to ECSs and user admin is used to access FusionInsight Manager.
- Password: Set a password for user root/admin.
- Confirm Password: Enter the password of user root/admin again.
- Select Enable to enable secure communications. For details, see Communication Security Authorization.
- Click Create Now.
If Kerberos Authentication is toggled on, the system asks you to confirm whether this function is required. If it is, click Continue. If not, click Back to disable it and then proceed with the subsequent steps. This option cannot be changed after you create a cluster.
+ - Click Back to Cluster List to view the cluster status. Click Access Cluster to view cluster details.
For details about cluster status during creation, see the description of the status parameters in Table 1.
+It takes some time to create a cluster. The initial status of the cluster is Starting. The cluster status will change to Running if the cluster is successfully created.
+On the MRS management console, a maximum of 10 clusters can be created concurrently, and a maximum of 100 clusters can be managed.
+
++ diff --git a/docs/mrs/umn/mrs_01_9003.html b/docs/mrs/umn/mrs_01_9003.html index bbd85c53..53636f61 100644 --- a/docs/mrs/umn/mrs_01_9003.html +++ b/docs/mrs/umn/mrs_01_9003.html @@ -8,7 +8,19 @@ -++Parent topic: Quick Creation of a Cluster+2024-01-25
+2024-08-02
+Modified the following content:
+- Fixed link errors in some sections.
- Added components and version information for MRS 3.3.0-LTS.1. For details, see List of MRS Component Versions.
- Added Creating a Doris Cluster. For details, see Quickly Creating a Doris Cluster
- Modified the descriptions of some parameters for configuring a cluster. For details, see Creating a Cluster, Quick Creation of a Hadoop Analysis Cluster, Quick Creation of an HBase Analysis Cluster, Quick Creation of a ClickHouse Cluster, Quick Creation of a Real-time Analysis Cluster, Creating a Custom Cluster and Creating a Custom Topology Cluster.
- Added descriptions about adding a tag to a cluster/node. For details, see Adding a Tag to a Cluster/Node.
- Modified descriptions about storage and compute decoupling, adjusted the sections, and added Guardian-based storage and compute decoupling. For details, see Configuring a Cluster with Storage and Compute Decoupled.
- The following functions are modified on FusionInsight Manager:
- The home page of Manager is changed. You can download the client and start or stop a cluster on the home page. For details, see Cluster Management.
- The monitoring indicator data report supports the change of the user-defined time length. For details, see Home Page.
- You can view the monitoring metric values in Additional Information if thresholds are set for the metrics to generate alarms. For details, see Alarms.
- When downloading logs on Manager, you can configure the Max. Concurrent Nodes. For details, see Log Download.
- Configure parameters on FusionInsight Manager. On some configuration pages, you do not need to manually select the current cluster name. For example, Adding a Tenant.
Added the following content:
+- Added the description of the principles of the Doris, Guardian and JobGateway components. For details, see Apache Doris, Guardian and JobGateway.
- The following functions are added to FusionInsight Manager:
- Supports quick DataNode decommissioning. For details, see Decommissioning and Recommissioning an Instance.
- Supports the configuration of hierarchical alarms. You can configure different alarm severities for different thresholds. For details, see Alarms.
- The Superior scheduler is used to schedule Yarn task queues support cross-pool scheduling. For details, see Configuring a Queue.
- Scheduler switchover for newly provisioned clusters is supported. For details, see Switching the Scheduler.
- Supports the configuration of SQL inspection. For details, see SQL Inspector.
- Supports the configuration of the Manager page timeout interval. For details, see Changing the Timeout Duration of the Manager Page.
Added the following alarms:
+- The Doris component is added to MRS 3.3.0-LTS.1, and alarms ALM-50201 Doris Service Unavailable~ALM-50226 BE Instance Fault are added.
- The Guardian component is added to MRS 3.3.0-LTS.1, and alarms ALM-45736 Guardian Service Unavailable~ALM-45743 Failed to Call the IAM API are added.
- ALM-12186 CGroup Task Usage Exceeds the Threshold
- ALM-12187 Failed to Expand Disk Partition Capacity
- ALM-12187 Failed to Expand Disk Partition Capacity
- ALM-14031 DataNode Process Is Abnormal
- ALM-14032 JournalNode Process Is Abnormal
- ALM-14033 ZKFC Process Is Abnormal
- ALM-14034 Router Process Is Abnormal
- ALM-14035 HttpFS Process Is Abnormal
- ALM-19022 HBase Hotspot Detection Is Unavailable
- ALM-19023 Region Traffic Restriction for HBase
- ALM-19024 RPC Requests P99 Latency on RegionServer Exceeds the Threshold
- ALM-19025 Damaged StoreFile in HBase
- ALM-19026 Damaged WAL Files in HBase
- ALM-25007 Number of SlapdServer Connections Exceeds the Threshold
- ALM-25008 SlapdServer CPU Usage Exceeds the Threshold
- ALM-29007 Impalad Process Memory Usage Exceeds the Threshold
- ALM-29008 Number of ODBC Connections to Impalad Exceeds the Threshold
- ALM-29010 Number of Queries Being Submitted by Impalad Exceeds the Threshold
- ALM-29011 Number of Queries Being Executed by Impalad Exceeds the Threshold
- ALM-29012 Number of Queries Being Waited by Impalad Exceeds the Threshold
- ALM-29013 Impalad FGC Time Exceeds the Threshold
- ALM-29014 Catalog FGC Time Exceeds the Threshold
- ALM-29015 Catalog Process Memory Usage Exceeds the Threshold
- ALM-29016 Impalad Instance in the Sub-healthy State
- ALM-45003 HetuEngine QAS Disk Capacity Is Insufficient
- ALM-45289 PolicySync Heap Memory Usage Exceeds the Threshold
- ALM-45290 PolicySync Direct Memory Usage Exceeds the Threshold
- ALM-45291 PolicySync Non-Heap Memory Usage Exceeds the Threshold
- ALM-45292 PolicySync GC Duration Exceeds the Threshold
- ALM-45435 Inconsistent Metadata of ClickHouse Tables
- ALM-45436 Skew ClickHouse Table Data
- ALM-45437 Excessive Parts in the ClickHouse Table
- ALM-45438 ClickHouse Disk Usage Exceeds 80%
- ALM-45439 ClickHouse Node Enters the Read-Only Mode
- ALM-45440 Inconsistency Between ClickHouse Replicas
- ALM-45441 Zookeeper Disconnected
- ALM-45442 Too Many Concurrent SQL Statements
- ALM-45443 Slow SQL Queries in the Cluster
- ALM-45444 Abnormal ClickHouse Process
- ALM-45642 RocksDB Continuously Triggers Write Traffic Limiting
- ALM-45643 MemTable Size of RocksDB Continuously Exceeds the Threshold
- ALM-45644 Number of SST Files at Level 0 of RocksDB Continuously Exceeds the Threshold
- ALM-45645 Pending Flush Size of RocksDB Continuously Exceeds the Threshold
- ALM-45646 Pending Compaction Size of RocksDB Continuously Exceeds the Threshold
- ALM-45647 Estimated Pending Compaction Size of RocksDB Continuously Exceeds the Threshold
- ALM-45648 RocksDB Frequently Encounters Write-Stopped
- ALM-45649 P95 Latency of RocksDB Get Requests Continuously Exceeds the Threshold
- ALM-45650 P95 Latency of RocksDB Write Requests Continuously Exceeds the Threshold
- ALM-45652 Flink Service Unavailable
- ALM-45653 Invalid Flink HA Certificate File
- ALM-45654 Flink HA Certificate Is About to Expire
- ALM-45655 Flink HA Certificate File Has Expired
- ALM-50401 Number of JobServer Jobs Waiting to Be Executed Exceeds the Threshold
- ALM-50402 JobGateway Service Unavailable
2024-01-25
Optimized the image format and modified the title description of MRS Manager.
2023-09-08
Modified the following content:
-- Updated available cluster versions. For details, see Creating a Custom Cluster.
- Added the description about system disk encryption parameters. For details, see Creating a Custom Cluster.
- Modified the constraints of adding a bootstrap action. For details, see Adding a Bootstrap Action.
- Add the description that the password of user omm is radomly generated. For details, see User Account ListUser Account ListSystem User Description.
- Optimized Flume alarm description. For details, see ALM-24010 Flume Certificate File Is Invalid or Damaged to ALM-24015 Flume MonitorServer Certificate File Has Expired.
- Updated available cluster versions. For details, see Creating a Custom Cluster.
- Added the description about system disk encryption parameters. For details, see Creating a Custom Cluster.
- Modified the constraints of adding a bootstrap action. For details, see Adding a Bootstrap Action.
- Add the description that the password of user omm is radomly generated. For details, see User Account List.
- Optimized Flume alarm description. For details, see ALM-24010 Flume Certificate File Is Invalid or Damaged to ALM-24015 Flume MonitorServer Certificate File Has Expired.
2023-07-27
@@ -168,5 +180,4 @@
- What Is MRS Used For?
- - What Types of Distributed Storage Does MRS Support?
- - How Do I Create an MRS Cluster Using a Custom Security Group?
- How Do I Use MRS?
@@ -52,9 +50,9 @@ - What Is the Compression Ratio of zstd?
- - CreatedWhy Are the HDFS, YARN, and MapReduce Components Unavailable When an MRS Cluster Is BoughtCreated?
+ - Why Are the HDFS, YARN, and MapReduce Components Unavailable When an MRS Cluster Is Created?
- - CreatedWhy Is the ZooKeeper Component Unavailable When an MRS Cluster Is BoughtCreated?
+ - Why Is the ZooKeeper Component Unavailable When an MRS Cluster Is Created?
- Which Python Versions Are Supported by Spark Tasks in an MRS 3.1.0 Cluster?
diff --git a/docs/mrs/umn/mrs_03_1153.html b/docs/mrs/umn/mrs_03_1153.html
index a9c3cfde..6d0b5df9 100644
--- a/docs/mrs/umn/mrs_03_1153.html
+++ b/docs/mrs/umn/mrs_03_1153.html
@@ -8,7 +8,6 @@
- Decompress the downloaded user credential file, and save the obtained krb5.conf and user.keytab files to the client directory, for example, /opt/client/Presto/.
- Run the following command to obtain the user principal:
- Run the following command to connect to the Presto Server of the cluster:
presto_cli.sh --krb5-config-path {krb5.conf file path} --krb5-principal {User's principal} --krb5-keytab-path {user.keytab file path} --user {presto username}
- krb5.conf file path: file path set in 3.e, for example, /opt/client/Presto/krb5.conf.
- user.keytab file path: file path set in 3.e, for example, /opt/client/Presto/user.keytab.
- User's principal: principal obtained in 3.f.
- presto username: user created in 3.b, for example, presto001.
Example: presto_cli.sh --krb5-config-path /opt/client/Presto/krb5.conf --krb5-principal prest001@xxx_xxx_xxx_xxx.COM --krb5-keytab-path /opt/client/Presto/user.keytab --user presto001
- On the Presto client, run the following statement to create a schema:
CREATE SCHEMA hive.demo01 WITH (location = 'obs://presto-demo002/');
- Create a table in the schema. The table data is stored in the OBS bucket, as shown in the following example:
CREATE TABLE hive.demo01.demo_table WITH (format = 'ORC') AS SELECT * FROM tpch.sf1.customer;
- Run exit to exit the client. diff --git a/docs/mrs/umn/mrs_03_1194.html b/docs/mrs/umn/mrs_03_1194.html index 438de8f5..3f9023c8 100644 --- a/docs/mrs/umn/mrs_03_1194.html +++ b/docs/mrs/umn/mrs_03_1194.html @@ -1,6 +1,6 @@ -
- Method 1: Replace the beeline -e " use default;show tables;" command with beeline --entirelineascommand=false -e "use default;show tables;".
- Method 2:
- In the /opt/Bigdata/client/Hive directory on the Hive client, change export CLIENT_HIVE_ENTIRELINEASCOMMAND=true in the component_env file to export CLIENT_HIVE_ENTIRELINEASCOMMAND=false.Figure 1 Changing the component_env filediff --git a/docs/mrs/umn/mrs_03_1202.html b/docs/mrs/umn/mrs_03_1202.html index 30af3a66..efc18c72 100644 --- a/docs/mrs/umn/mrs_03_1202.html +++ b/docs/mrs/umn/mrs_03_1202.html @@ -1,6 +1,6 @@ -
CreatedWhy Are the HDFS, YARN, and MapReduce Components Unavailable When an MRS Cluster Is BoughtCreated?
+Why Are the HDFS, YARN, and MapReduce Components Unavailable When an MRS Cluster Is Created?
The HDFS, YARN, and MapReduce components are integrated in Hadoop. If the three components are unavailable when are MRS cluster is created, select Hadoop instead. After an MRS cluster is created, HDFS, YARN, and MapReduce are available in the Components page.
diff --git a/docs/mrs/umn/mrs_03_1204.html b/docs/mrs/umn/mrs_03_1204.html index 58f34514..ccbde237 100644 --- a/docs/mrs/umn/mrs_03_1204.html +++ b/docs/mrs/umn/mrs_03_1204.html @@ -1,6 +1,6 @@ -CreatedWhy Is the ZooKeeper Component Unavailable When an MRS Cluster Is BoughtCreated?
+Why Is the ZooKeeper Component Unavailable When an MRS Cluster Is Created?
If you create a cluster of a version earlier than MRS 3.x, ZooKeeper is installed by default and is not displayed on the GUI.
If you create a cluster of MRS 3.x or later, ZooKeeper is available on the GUI and is selected by default.
After the cluster is created, the ZooKeeper component is available on the Components page.
diff --git a/docs/mrs/umn/mrs_03_2014.html b/docs/mrs/umn/mrs_03_2014.html index 3f46d9fa..eb7c067a 100644 --- a/docs/mrs/umn/mrs_03_2014.html +++ b/docs/mrs/umn/mrs_03_2014.html @@ -38,7 +38,7 @@- Can I Export the Query Result of Hive Data?
-
- beeline -eHow Do I Do If an Error Occurs When Hive Runs the beeline -e Command to Execute Multiple Statements?
+- How Do I Do If an Error Occurs When Hive Runs the beeline -e Command to Execute Multiple Statements?
- How Do I Do If a "hivesql/hivescript" Job Fails to Submit After Hive Is Added?
diff --git a/docs/mrs/umn/mrs_08_0005.html b/docs/mrs/umn/mrs_08_0005.html index ee16f748..39fdb048 100644 --- a/docs/mrs/umn/mrs_08_0005.html +++ b/docs/mrs/umn/mrs_08_0005.html @@ -5,236 +5,329 @@
- - Hadoop includes HDFS, YARN, and MapReduce.
- DBService, ZooKeeper, KrbServer, and LdapServer are components used in the cluster and are not displayed during cluster creation.
- MRS component versions must be consistent with open-source component versions.
Table 1 MRS component versions Component
+Table 1 MRS component versions Component
MRS 3.1.2-LTS.6
+MRS 3.1.2-LTS.6
MRS 3.2.0-LTS.2
+MRS 3.2.0-LTS.2
+3.3.0-LTS.1
+ -
+-
-
+-
+-
+ 2.2.0
+2.2.0
2.2.0
+2.2.0
+2.2.0
+ -
+-
1.0.0
+1.0.0
+-
+ 21.3.4.25
+21.3.4.25
22.3.2.2
+22.3.2.2
+23.3.2.37
+ 2.7.0
+2.7.0
2.7.0
+2.7.0
+2.7.0
+ 1.12.2
+-
1.15.0
+-
+1.2.3
+ 1.9.0
+1.12.2
1.9.0
+1.15.0
+1.15.0
+ 2.2.3
+1.9.0
2.2.3
+1.9.0
+1.11.0
+ 3.1.1
+-
3.3.1
+-
+0.1.0
+ 1.2.0
+2.2.3
1.2.0
+2.2.3
+2.4.14
+ 3.1.0
+3.1.1
3.1.0
+3.3.1
+3.3.1
+ 0.9.0
+1.2.0
0.11.0
+1.2.0
+2.0.0
+ 4.7.0
+3.1.0
4.7.0
+3.1.0
+3.1.0
+ -
+0.9.0
0.14.0
+0.11.0
+0.11.0
+ 2.11-2.4.0
+4.7.0
2.11-2.4.0
+4.7.0
+4.7.0
+ -
+-
-
+0.14.0
+-
+ 1.18
+-
1.18
+-
+1.1.0
+ 2.7.0
+2.11-2.4.0
2.7.0
+2.11-2.4.0
+2.12-2.8.1
+ 1.99.3
+-
1.99.3
+-
+-
+ 3.1.1
+1.18
3.3.1
+1.18
+1.20
+ 5.1.0
+2.7.0
5.1.0
+2.7.0
+2.7.0
+ -
+1.99.3
-
+1.99.3
+1.99.3
+ -
+3.1.1
-
+3.3.1
+3.3.1
Phoenix (integrated in HBase)
+5.0.0
+5.1.0
5.0.0
+5.1.0
+5.10
+ 2.0.0
+-
2.0.0
+-
+-
+ -
+-
-
+-
+-
+ Phoenix (integrated in HBase)
3.1.1
+5.0.0
3.1.1
+5.0.0
+5.1.2
+ -
+2.0.0
-
+2.0.0
+2.3.0
+ 0.9.2
+-
0.9.2
+-
+3.3.1
+ 3.1.1
+3.1.1
3.3.1
+3.1.1
+-
+ 3.6.3
+-
3.6.3
+-
+-
+ 8.1.2
+0.9.2
8.2.0.1
+0.9.2
+0.10.2
++ +3.1.1
+3.3.1
+3.3.1
++ +3.6.3
+3.6.3
+3.8.1
++ +8.1.2
+8.2.0.1
+8.3.0.1
The Spark component applies to versions earlier than MRS 3.x.
Description
Spark is an open source parallel data processing framework. It helps you easily develop unified big data applications and perform offline processing, stream processing, and interactive analysis on data.
-Spark provides a framework featuring fast computing, write, and interactive query. Spark has obvious advantages over Hadoop in terms of performance. Spark uses the in-memory computing mode to avoid I/O bottlenecks in scenarios where multiple tasks in a MapReduce workflow process the same dataset. Spark is implemented by using Scala programming language. Scala enables distributed datasets to be processed in a method that is the same as that of processing local data. In addition to interactive data analysis, Spark supports interactive data mining. Spark adopts in-memory computing, which facilitates iterative computing. By coincidence, iterative computing of the same data is a general problem facing data mining. In addition, Spark can run in Yarn clusters where Hadoop 2.0 is installed. The reason why Spark cannot only retain various features like MapReduce fault tolerance, data localization, and scalability but also ensure high performance and avoid busy disk I/Os is that a memory abstraction structure called Resilient Distributed Dataset (RDD) is created for Spark.
+Spark provides a framework featuring fast computing, write, and interactive query. Spark has obvious advantages over Hadoop in terms of performance. Spark uses the in-memory computing mode to avoid I/O bottlenecks in scenarios where multiple tasks in a MapReduce workflow process the same dataset. Spark is implemented by using Scala programming language. Scala enables distributed datasets to be processed in a method that is the same as that of processing local data. In addition to interactive data analysis, Spark supports interactive data mining. Spark adopts in-memory computing, which facilitates iterative computing. By coincidence, iterative computing of the same data is a general problem facing data mining. In addition, Spark can run in Yarn clusters where Hadoop 2.0 is installed. The reason why Spark cannot only retain various features like MapReduce fault tolerance, data localization, and scalability but also ensure high performance and avoid busy disk I/Os is that a memory abstraction structure called Resilient Distributed Dataset (RDD) is created for Spark.
Original distributed memory abstraction, for example, key-value store and databases, supports small-granularity update of variable status. This requires backup of data or log updates to ensure fault tolerance. Consequently, a large amount of I/O consumption is brought about to data-intensive workflows. For the RDD, it has only one set of restricted APIs and only supports large-granularity update, for example, map and join. In this way, Spark only needs to record the transformation operation logs generated during data establishment to ensure fault tolerance without recording a complete dataset. This data transformation link record is a source for tracing a data set. Generally, parallel applications apply the same computing process for a large dataset. Therefore, the limit to the mentioned large-granularity update is not large. As described in Spark theses, the RDD can function as multiple different computing frameworks, for example, programming models of MapReduce and Pregel. In addition, Spark allows you to explicitly make a data transformation process be persistent on hard disks. Data localization is implemented by allowing you to control data partitions based on the key value of each record. (An obvious advantage of this method is that two copies of data to be associated will be hashed in the same mode.) If memory usage exceeds the physical limit, Spark writes relatively large partitions into hard disks, thereby ensuring scalability.
Spark has the following features:
- Fast: The data processing speed of Spark is 10 to 100 times higher than that of MapReduce.
- Easy-to-use: Java, Scala, and Python can be used to simply and quickly compile parallel applications for processing massive amounts of data. Spark provides over 80 operators to help you compile parallel applications.
- Universal: Spark provides many tools, for example, Spark SQL and Spark Streaming. These tools can be combined flexibly in an application.
- Integration with Hadoop: Spark can directly run in a Hadoop cluster and read existing Hadoop data.
The Spark component of MRS has the following advantages:
- The Spark Streaming component of MRS supports real-time data processing rather than triggering as scheduled.
- The Spark component of MRS provides Structured Streaming and allows you to build streaming applications using the Dataset API. Spark supports exactly-once semantics and inner and outer joins for streams.
- The Spark component of MRS uses pandas_udf to replace the original user-defined functions (UDFs) in PySpark to process data, which reduces the processing duration by 60% to 90% (affected by specific operations).
- The Spark component of MRS also supports graph data processing and allows modeling using graphs during graph computing.
- Spark SQL of MRS is compatible with some Hive syntax (based on the 64 SQL statements of the Hive-Test-benchmark test set) and standard SQL syntax (based on the 99 SQL statements of the TPC-DS test set).
For details about Spark architecture and principles, visit https://spark.apache.org/docs/3.1.1/quick-start.html.
+For details about Spark architecture and principles, visit https://archive.apache.org/dist/spark/docs/3.1.1/quick-start.html.
Architecture
Figure 1 describes the Spark architecture and Table 1 lists the Spark modules.
@@ -24,7 +24,7 @@Cluster Manager
Cluster manager manages resources in the cluster. Spark supports multiple cluster managers, including Mesos, Yarn, and the Standalone cluster manager that is delivered with Spark.
+Cluster manager manages resources in the cluster. Spark supports multiple cluster managers, including Mesos, Yarn, and the Standalone cluster manager that is delivered with Spark.
Application
@@ -59,7 +59,7 @@Stage
Each job consists of multiple stages. Each stage is a task set, which is separated by Directed Acyclic Graph (DAG).
+Each job consists of multiple stages. Each stage is a task set, which is separated by Directed Acyclic Graph (DAG).
Task
@@ -72,7 +72,7 @@Spark Application Running Principle
Figure 2 shows the Spark application running architecture. The running process is as follows:
-- An application is running in the cluster as a collection of processes. Driver coordinates the running of the application.
- To run an application, Driver connects to the cluster manager (such as Standalone, Mesos, and Yarn) to apply for the executor resources, and start ExecutorBackend. The cluster manager schedules resources between different applications. Driver schedules DAGs, divides stages, and generates tasks for the application at the same time.
- Then, Spark sends the codes of the application (the codes transferred to SparkContext, which is defined by JAR or Python) to an executor.
- After all tasks are finished, the running of the user application is stopped.
- An application is running in the cluster as a collection of processes. Driver coordinates the running of the application.
- To run an application, Driver connects to the cluster manager (such as Standalone, Mesos, and Yarn) to apply for the executor resources, and start ExecutorBackend. The cluster manager schedules resources between different applications. Driver schedules DAGs, divides stages, and generates tasks for the application at the same time.
- Then, Spark sends the codes of the application (the codes transferred to SparkContext, which is defined by JAR or Python) to an executor.
- After all tasks are finished, the running of the user application is stopped.
Figure 3 shows the Master and Worker modes adopted by Spark. A user submits an application on the Spark client, and then the scheduler divides a job into multiple tasks and sends the tasks to each Worker for execution. Each Worker reports the computation results to Driver (Master), and then the Driver aggregates and returns the results to the client.
@@ -83,9 +83,9 @@- In Yarn-cluster mode, the Spark driver runs inside an ApplicationMaster process which is managed by Yarn in the cluster. After the ApplicationMaster is started, the client can exit without interrupting service running.
- In Yarn-client mode, the driver is started in the client process, and the ApplicationMaster process is used only to apply for resources from the Yarn cluster.
Spark Streaming Principle
Spark Streaming is a real-time computing framework built on the Spark, which expands the capability for processing massive streaming data. Currently, Spark supports the following data processing methods:
-- Direct Streaming
In Direct Streaming approach, Direct API is used to process data. Take Kafka Direct API as an example. Direct API provides offset location that each batch range will read from, which is much simpler than starting a receiver to continuously receive data from Kafka and written data to write-ahead logs (WALs). Then, each batch job is running and the corresponding offset data is ready in Kafka. These offset information can be securely stored in the checkpoint file and read by applications that failed to start.
+- Direct Streaming
In Direct Streaming approach, Direct API is used to process data. Take Kafka Direct API as an example. Direct API provides offset location that each batch range will read from, which is much simpler than starting a receiver to continuously receive data from Kafka and written data to write-ahead logs (WALs). Then, each batch job is running and the corresponding offset data is ready in Kafka. These offset information can be securely stored in the checkpoint file and read by applications that failed to start.
Figure 4 Data transmission through Direct Kafka API-
After the failure, Spark Streaming can read data from Kafka again and process the data segment. The processing result is the same no matter Spark Streaming fails or not, because the semantic is processed only once.
+After the failure, Spark Streaming can read data from Kafka again and process the data segment. The processing result is the same no matter Spark Streaming fails or not, because the semantic is processed only once.
Direct API does not need to use the WAL and Receivers, and ensures that each Kafka record is received only once, which is more efficient. In this way, the Spark Streaming and Kafka can be well integrated, making streaming channels be featured with high fault-tolerance, high efficiency, and ease-of-use. Therefore, you are advised to use Direct Streaming to process data.
- Receiver
When a Spark Streaming application starts (that is, when the driver starts), the related StreamingContext (the basis of all streaming functions) uses SparkContext to start the receiver to become a long-term running task. These receivers receive and save streaming data to the Spark memory for processing. Figure 5 shows the data transfer lifecycle.
@@ -128,7 +128,7 @@Spark 3.1.1 realizes command localization and replaces the Hive with Spark SQL Native DDL/DML to run DDL/DML commands. Additionally, the decoupling from the Hive is realized and commands can be customized.
DataSet
A DataSet is a strongly typed collection of domain-specific objects that can be transformed in parallel using functional or relational operations. Each Dataset also has an untyped view called a DataFrame, which is a Dataset of Row.
-The DataFrame is a structured and distributed dataset consisting of multiple columns. The DataFrame is equal to a table in the relationship database or the DataFrame in the R/Python. The DataFrame is the most basic concept in the Spark SQL, which can be created by using multiple methods, such as the structured dataset, Hive table, external database or RDD.
+The DataFrame is a structured and distributed dataset consisting of multiple columns. The DataFrame is equal to a table in the relationship database or the DataFrame in the R/Python. The DataFrame is the most basic concept in the Spark SQL, which can be created by using multiple methods, such as the structured dataset, Hive table, external database or RDD.
Operations available on DataSets are divided into transformations and actions.
- A transformation operation can generate a new DataSet,
- An action operation can trigger computation and return results,
for example, count, show, or write data to the file system.
@@ -214,7 +214,7 @@ errors.count()SparkHome: indicates the directory where Spark is installed in the cluster.
jars: indicates the code and dependency package of an application.
- Main program: processes data.
For details about how to submit an application, visit https://spark.apache.org/docs/3.1.1/submitting-applications.html.
+For details about how to submit an application, visit https://spark.apache.org/docs/3.1.1/submitting-applications.html.
- Spark Shell Commands
The basic Spark shell commands support the submission of Spark applications. The Spark shell commands are as follows:
./bin/spark-submit \ --class <main-class> \ diff --git a/docs/mrs/umn/mrs_08_0052.html b/docs/mrs/umn/mrs_08_0052.html index be87e4a5..f3fc7fee 100644 --- a/docs/mrs/umn/mrs_08_0052.html +++ b/docs/mrs/umn/mrs_08_0052.html @@ -16,10 +16,14 @@
- DBService
+ - Apache Doris
+ - Flink
- Flume
+ - Guardian
+ - HBase
- HDFS
@@ -34,6 +38,8 @@ - IoTDB
+ - JobGateway
+ - Kafka
- KafkaManager
diff --git a/docs/mrs/umn/mrs_08_007101.html b/docs/mrs/umn/mrs_08_007101.html index 65cd02d7..1e362eb4 100644 --- a/docs/mrs/umn/mrs_08_007101.html +++ b/docs/mrs/umn/mrs_08_007101.html @@ -53,7 +53,7 @@
- Direct Streaming
Stage
Each job consists of multiple stages. Each stage is a task set, which is separated by Directed Acyclic Graph (DAG).
+Each job consists of multiple stages. Each stage is a task set, which is separated by Directed Acyclic Graph (DAG).
Task
@@ -66,7 +66,7 @@Spark Principle
Figure 2 describes the application running architecture of Spark.
-- An application is running in the cluster as a collection of processes. Driver coordinates the running of the application.
- To run an application, Driver connects to the cluster manager (such as Standalone, Mesos, and Yarn) to apply for the executor resources, and start ExecutorBackend. The cluster manager schedules resources between different applications. Driver schedules DAGs, divides stages, and generates tasks for the application at the same time.
- Then, Spark sends the codes of the application (the codes transferred to SparkContext, which is defined by JAR or Python) to an executor.
- After all tasks are finished, the running of the user application is stopped.
- An application is running in the cluster as a collection of processes. Driver coordinates the running of the application.
- To run an application, Driver connects to the cluster manager (such as Standalone, Mesos, and Yarn) to apply for the executor resources, and start ExecutorBackend. The cluster manager schedules resources between different applications. Driver schedules DAGs, divides stages, and generates tasks for the application at the same time.
- Then, Spark sends the codes of the application (the codes transferred to SparkContext, which is defined by JAR or Python) to an executor.
- After all tasks are finished, the running of the user application is stopped.
Spark uses Master and Worker modes, as shown in Figure 3. A user submits an application on the Spark client, and then the scheduler divides a job into multiple tasks and sends the tasks to each Worker for execution. Each Worker reports the computation results to Driver (Master), and then the Driver aggregates and returns the results to the client.
@@ -78,9 +78,9 @@Spark Streaming Principle
Spark Streaming is a real-time computing framework built on the Spark, which expands the capability for processing massive streaming data. Spark supports two data processing approaches: Direct Streaming and Receiver.
Direct Streaming computing process
-In Direct Streaming approach, Direct API is used to process data. Take Kafka Direct API as an example. Direct API provides offset location that each batch range will read from, which is much simpler than starting a receiver to continuously receive data from Kafka and written data to write-ahead logs (WALs). Then, each batch job is running and the corresponding offset data is ready in Kafka. These offset information can be securely stored in the checkpoint file and read by applications that failed to start.
+In Direct Streaming approach, Direct API is used to process data. Take Kafka Direct API as an example. Direct API provides offset location that each batch range will read from, which is much simpler than starting a receiver to continuously receive data from Kafka and written data to write-ahead logs (WALs). Then, each batch job is running and the corresponding offset data is ready in Kafka. These offset information can be securely stored in the checkpoint file and read by applications that failed to start.
Figure 4 Data transmission through Direct Kafka API-
After the failure, Spark Streaming can read data from Kafka again and process the data segment. The processing result is the same no matter Spark Streaming fails or not, because the semantic is processed only once.
+After the failure, Spark Streaming can read data from Kafka again and process the data segment. The processing result is the same no matter Spark Streaming fails or not, because the semantic is processed only once.
Direct API does not need to use the WAL and Receivers, and ensures that each Kafka record is received only once, which is more efficient. In this way, the Spark Streaming and Kafka can be well integrated, making streaming channels be featured with high fault-tolerance, high efficiency, and ease-of-use. Therefore, you are advised to use Direct Streaming to process data.
Receiver computing process
When a Spark Streaming application starts (that is, when the driver starts), the related StreamingContext (the basis of all streaming functions) uses SparkContext to start the receiver to become a long-term running task. These receivers receive and save streaming data to the Spark memory for processing. Figure 5 shows the data transfer lifecycle.
@@ -122,7 +122,7 @@Spark2x realizes command localization and replaces the Hive with Spark SQL Native DDL/DML to run DDL/DML commands. Additionally, the decoupling from the Hive is realized and commands can be customized.
DataSet
A DataSet is a strongly typed collection of domain-specific objects that can be transformed in parallel using functional or relational operations. Each Dataset also has an untyped view called a DataFrame, which is a Dataset of Row.
-The DataFrame is a structured and distributed dataset consisting of multiple columns. The DataFrame is equal to a table in the relationship database or the DataFrame in the R/Python. The DataFrame is the most basic concept in the Spark SQL, which can be created by using multiple methods, such as the structured dataset, Hive table, external database or RDD.
+The DataFrame is a structured and distributed dataset consisting of multiple columns. The DataFrame is equal to a table in the relationship database or the DataFrame in the R/Python. The DataFrame is the most basic concept in the Spark SQL, which can be created by using multiple methods, such as the structured dataset, Hive table, external database or RDD.
Operations available on DataSets are divided into transformations and actions.
- A transformation operation can generate a new DataSet,
- An action operation can trigger computation and return results,
for example, count, show, or write data to the file system.
@@ -208,7 +208,7 @@ errors.count()SparkHome: indicates the directory where Spark is installed in the cluster.
jars: indicates the code and dependency package of an application.
- Main program: processes data.
For details about how to submit an application, visit https://spark.apache.org/docs/3.1.1/submitting-applications.html.
+For details about how to submit an application, visit https://archive.apache.org/dist/spark/docs/3.1.1/submitting-applications.html.
- Spark Shell Commands
The basic Spark shell commands support the submission of Spark applications. The Spark shell commands are as follows:
./bin/spark-submit \ --class <main-class> \ diff --git a/docs/mrs/umn/mrs_08_0149.html b/docs/mrs/umn/mrs_08_0149.html new file mode 100644 index 00000000..f36708fe --- /dev/null +++ b/docs/mrs/umn/mrs_08_0149.html @@ -0,0 +1,18 @@ + + +
JobGateway
++++ + ++ diff --git a/docs/mrs/umn/mrs_08_0150.html b/docs/mrs/umn/mrs_08_0150.html new file mode 100644 index 00000000..fe84dbcb --- /dev/null +++ b/docs/mrs/umn/mrs_08_0150.html @@ -0,0 +1,16 @@ + + +++Parent topic: Components+JobGateway Basic Principles
++JobGateway allows you to submit jobs through REST APIs.
+As a gateway component for submitting big data jobs, JobGateway provides fully controllable enterprise-level big data job submission services, such as Spark, HBase, Flink, and Hive.
++JobGateway Architecture
JobGateway consists of JobServer and JobBalancer instances.
+- JobBalancer provides load balancing.
- JobServer provides REST APIs for submitting jobs.
Figure 1 JobGateway architecture+
++ diff --git a/docs/mrs/umn/mrs_08_0152.html b/docs/mrs/umn/mrs_08_0152.html new file mode 100644 index 00000000..b4edff67 --- /dev/null +++ b/docs/mrs/umn/mrs_08_0152.html @@ -0,0 +1,12 @@ + + +++Parent topic: JobGateway+Relationships Between JobGateway and Other Components
++JobGateway is a service that allows you to submit Spark, Hive, MapReduce, and Flink jobs through REST APIs.
+Figure 1 Relationships between JobGateway and other components+
++ diff --git a/docs/mrs/umn/mrs_08_0159.html b/docs/mrs/umn/mrs_08_0159.html new file mode 100644 index 00000000..8e51ae2f --- /dev/null +++ b/docs/mrs/umn/mrs_08_0159.html @@ -0,0 +1,21 @@ + + +++Parent topic: JobGateway+Guardian
+++Guardian Basic Principles
Guardian is a service that provides temporary authentication credentials for services such as HDFS, Hive, Spark, HBase, Loader and HetuEngine to access OBS in decoupled storage and compute scenarios. The Guardian component needs to be installed only when OBS is connected. Typical features of Guardian include:+- Provides the capability of obtaining temporary authentication credentials for accessing OBS.
- Provides fine-grained permission control for accessing OBS.
- Provides the unified cache refreshing capability for temporary authentication credentials used to access OBS.
The Guardian server provides functions for the TokenServer role. TokenServer supports multi-instance deployment. Each instance can have the same functions. A single point of failure (SPOF) does not affect service functions. In addition, the Guardian server provides RPC and HTTPS interfaces to obtain temporary authentication credentials for accessing OBS.
++Guardian Architecture
Figure 1 shows the basic architecture of Guardian.
+ ++ +Relationships Between Guardian and Other Components
Before accessing OBS, HDFS, Hive, Spark, Flink, HBase, Loader, and HetuEngine access Guardian to obtain temporary credentials for the access. Guardian generates a temporary credential with fine-grained authentication content based on the IAM access request of the current login user and returns the credential to the component. The component uses the credential to access OBS. OBS determines whether the current user has the access permission based on the credential.
+Figure 2 Relationships between Guardian and other components+
++ diff --git a/docs/mrs/umn/mrs_08_0161.html b/docs/mrs/umn/mrs_08_0161.html new file mode 100644 index 00000000..d8ffdb8b --- /dev/null +++ b/docs/mrs/umn/mrs_08_0161.html @@ -0,0 +1,17 @@ + + +++Parent topic: Components+Apache Doris
+ ++ + ++ diff --git a/docs/mrs/umn/mrs_08_0162.html b/docs/mrs/umn/mrs_08_0162.html new file mode 100644 index 00000000..ef020164 --- /dev/null +++ b/docs/mrs/umn/mrs_08_0162.html @@ -0,0 +1,75 @@ + + +++Parent topic: Components+Basic Principles
+++Introduction to Doris
Doris is a high-performance, real-time analytical database based on MPP architecture, known for its extreme speed and ease of use. It can return query results of mass data in sub-seconds and can support high-concurrency point queries and high-throughput complex analysis. All this makes Apache Doris an ideal tool for report analysis, ad-hoc query, unified data warehouse, and data lake query acceleration. On Doris, users can build various applications, such as user behavior analysis, AB test platform, log retrieval analysis, user portrait analysis, and order analysis. For more information, see Apache Doris.
++Doris Architecture
The following figure shows the overall architecture of Doris. The frontend (FE) and backend (BE) nodes can be expanded horizontally and infinitely.
+Figure 1 Doris architecture+ +
+
+Table 1 Description Parameter
+Description
+MySQL Tools
+Doris is fully compatible with MySQL syntax and can be accessed by various client tools. It also supports standard SQL statements and can seamlessly connect to BI tools.
+FE
+Frontend nodes process user access requests, plan query parsing, and manage metadata and nodes.
+BE
+Backend nodes store data, execute query plans, and balance load among copies.
+Leader
+Leader is a role elected from Follower nodes.
+Follower
+Follower nodes receive metadata logs, which must be written successfully in most nodes.
+Doris uses the MPP model for inter-node and intra-node parallel execution, making it suitable for distributed joins of large tables.
+It also supports vectorized query execution engines, adaptive query execution (AQE) technology, optimization strategies that combine CBO and RBO, and hot data cache queries.
++Basic Concepts
In Doris, data is logically described in the form of tables.
+- Rows and Columns
A table consists of rows and columns.
+- Row: a row of user data.
- Column: different fields in a row of data.
Columns can be classified into two types: keys and values. From the service perspective, Key and Value correspond to dimension columns and metric columns, respectively. In the aggregation model, rows with the same Key column are aggregated into one row. How Value columns are aggregated is specified by a user when the table is created.
+ - Tablets and Partitions
In the Doris storage engine, user data is horizontally divided into several tablets (also called data buckets). Each tablet contains several rows of data. The data between the individual tablets does not intersect and is physically stored independently.
+Multiple tablet logically belong to different partitions. A tablet belongs to only one partition, but a partition can contain multiple tablets. Since the tablets are physically stored independently, the partitions can be seen as physically independent, too. Tablet is the smallest physical storage unit for data operations such as movement and replication.
+Multiple partitions form a table. A partition can be regarded as the smallest logical unit for management. Data can be imported or deleted only for one partition.
+ - Data Models
Doris data models are classified into three types: Aggregate, Unique, and Duplicate.
+- Aggregate Model
When data is imported, rows with the same Key column are aggregated, and the Value columns are aggregated based on the AggregationType configured by users. AggregationType has the following modes:
+- SUM: Sum up the values in multiple rows.
- REPLACE: Replace the previous value with the newly imported value.
- MAX: Keep the maximum value.
- MIN: Keep the minimum value.
- Unique Model
In some multi-dimensional analysis scenarios, users are highly concerned about how to create uniqueness constraints for the Primary Key. The Unique model is introduced to solve this problem.
+- Merge on Read
The merge on read implementation in the Unique model is equivalent to Replace implementation in the Aggregate model. The internal implementation and data storage method are the same.
+ - Merge on Write
The Merge on Write implementation of the Unique model is completely different from that of the Aggregate model. It can deliver better performance (almost like that of the Duplicate model) in aggregation queries with primary key limitations. This implementation is particularly suitable for aggregation queries and those using indexes to filter out large scale data.
+In a Unique table where Merge on Write is enabled, overwritten and updated data is marked and deleted during data import, and new data is written to a new file. During a query, all data marked for deletion is filtered out at the file level, and the read data is the latest data. This eliminates the data aggregation process in Merge on Read and supports pushdown of multiple predicates in many cases. Performance can be greatly improved in many scenarios, especially in the case of aggregation queries.
+
- Merge on Read
- Duplicate Model
In some multi-dimensional analysis scenarios, primary keys and data aggregation are not required. Duplicate models can be introduced to meet such requirements.
+Different from the Aggregate and Unique models, the Duplicate model stores the data as they are and executes no aggregation. Even if there are two identical rows of data, they will both be retained. The DUPLICATE KEY in the CREATE TABLE statement is only used to specify based on which columns the data are sorted.
+ - Data Model Selection
The data model is established when the table is created and cannot be modified. Therefore, it is important to select a proper data model.
+- The Aggregate model aggregates data in advance, greatly reducing data scanning and calculation workload. Therefore, it is suitable for reporting query business, which has fixed schema. However, this model is not user-friendly for count(*) queries. Since the aggregation method on the Value column is fixed, semantic correctness should be considered in other types of aggregation queries.
- The Unique model ensures that the primary key is unique when it is required. However, pre-aggregation such as Rollup cannot be used in this case.
The Unique model supports only the update of an entire row. If you need to update both the unique primary key constraint and some columns (for example, importing multiple source tables to one Doris table), you can use the Aggregate model and set the aggregation type of non-primary key columns to REPLACE_IF_NOT_NULL.
+ - Duplicate is suitable for ad-hoc queries in any dimension. Although pre-aggregation cannot be used, Duplicate is not restricted by the aggregation model and can make full use of the advantages of the column-store model, that is, only related columns are read, and not all key columns need to be read.
- Aggregate Model
++ diff --git a/docs/mrs/umn/mrs_08_0163.html b/docs/mrs/umn/mrs_08_0163.html new file mode 100644 index 00000000..1f8f4ce0 --- /dev/null +++ b/docs/mrs/umn/mrs_08_0163.html @@ -0,0 +1,22 @@ + + +++Parent topic: Apache Doris+Relationship with Other Components
+++HDFS
Doris can import and export HDFS data and directly query HDFS data sources.
++Hudi
Doris can directly query Hudi data sources.
++Spark
Spark Doris Connector allows Spark to read data stored in Doris and write data to Doris.
++Flink
Flink Doris Connector allows you to perform operations (read, insert, modify, and delete) on data stored in Doris through Flink.
++Hive
Doris can directly query Hive data sources.
++Kafka
Doris can import Kafka data.
+++++Parent topic: Apache Doris+ - Can I Export the Query Result of Hive Data?
- In the /opt/Bigdata/client/Hive directory on the Hive client, change export CLIENT_HIVE_ENTIRELINEASCOMMAND=true in the component_env file to export CLIENT_HIVE_ENTIRELINEASCOMMAND=false.
beeline -eHow Do I Do If an Error Occurs When Hive Runs the beeline -e Command to Execute Multiple Statements?
+How Do I Do If an Error Occurs When Hive Runs the beeline -e Command to Execute Multiple Statements?
When Hive of MRS 3.x runs the beeline -e " use default;show tables;" command, the following error message is displayed: Error while compiling statement: FAILED: ParseException line 1:11 missing EOF at ';' near 'default' (state=42000,code=40000).
Solutions:
- Service: Click
- If the alarm sending switch is displayed as
