A200161411/cloud-container-engine
1
0
Fork 0
forked from docs/cloud-container-engine
Code Pull Requests Activity
cloud-container-engine/umn/source/add-ons/npd.rst

134 KiB
Raw Blame History

original_name

cce_10_0132.html

npd

Introduction

node-problem-detector (npd for short) is an add-on that monitors abnormal events of cluster nodes and connects to a third-party monitoring platform. It is a daemon running on each node. It collects node issues from different daemons and reports them to the API server. The npd add-on can run as a DaemonSet or a daemon.

For more information, see node-problem-detector.

Notes and Constraints

  • When using this add-on, do not format or partition node disks.
  • Each npd process occupies 30 mCPU and 100 MB memory.

Permission Description

To monitor kernel logs, the npd add-on needs to read the host /dev/kmsg. Therefore, the privileged mode must be enabled. For details, see privileged.

In addition, CCE mitigates risks according to the least privilege principle. Only the following privileges are available for npd running:

  • cap_dac_read_search: permission to access /run/log/journal.
  • cap_sys_admin: permission to access /dev/kmsg.

Installing the Add-on

  1. Log in to the CCE console, click the cluster name, and access the cluster console. Choose Add-ons in the navigation pane, locate npd on the right, and click Install.

  2. On the Install Add-on page, select the add-on specifications and set related parameters.

    • Pods: Set the number of pods based on service requirements.
    • Containers: Select a proper container quota based on service requirements.

  3. Set the parameters according to the following table and click Install.

    Only 1.16.0 and later versions support the configurations.

    npc.enable: indicates whether to enable Node-problem-controller <cce_10_0132__section1471610580474>.

npd Check Items

Note

Check items are supported only in 1.16.0 and later versions.

Check items cover events and statuses.

  • Event-related

    For event-related check items, when a problem occurs, npd reports an event to the API server. The event type can be Normal (normal event) or Warning (abnormal event).

    Table 1 Event-related check items
    Check Item Function Description
    OOMKilling Check whether OOM events occur and are reported. Warning event
    TaskHung Check whether taskHung events occur and are reported. Warning event
    KernelOops Check kernel nil pointer panic errors. Warning event
    ConntrackFull Check whether the conntrack table is full.

    Warning event

    Interval: 30 seconds

    Threshold: 80%

  • Status-related

    For status-related check items, when a problem occurs, npd reports an event to the API server and changes the node status synchronously. This function can be used together with Node-problem-controller fault isolation <cce_10_0132__section1471610580474> to isolate nodes.

    If the check period is not specified in the following check items, the default period is 30 seconds.

    Table 2 Application and OS check items
    Check Item Function Description
    FrequentKubeletRestart Check whether kubelet restarts frequently by listening to journald logs.

    • Interval: 5 minutes

    • Backtracking: 10 minutes

    • Threshold: 10 times

      If the system restarts for 10 times within the backtracking period, it indicates that the system restarts frequently and a fault alarm is generated.

    • Listening object: logs in the /run/log/journal directory

    Note

    The Ubuntu OS does not support the preceding check items due to incompatible log formats.
    FrequentDockerRestart Check whether Docker restarts frequently by listening to journald logs.
    FrequentContainerdRestart Check whether containerd restarts frequently by listening to journald logs.
    CRIProblem Check the CRI component status. Check object: Docker or containerd
    KUBELETProblem Check the kubelet status. None
    NTPProblem

    Check the NTP and Chrony service status.

    Check whether the node clock offsets.

    		 Threshold of the clock offset: 8000 ms
    PIDProblem Check whether PIDs are sufficient.
    • Threshold: 90%
    • Usage: nr_threads in /proc/loadavg
    • Maximum value: smaller value between /proc/sys/kernel/pid_max and /proc/sys/kernel/threads-max.
    FDProblem Check whether file handles are sufficient.
    • Threshold: 90%
    • Usage: the first value in /proc/sys/fs/file-nr
    • Maximum value: the third value in /proc/sys/fs/file-nr
    MemoryProblem Check whether the overall node memory is sufficient.
    • Threshold: 90%
    • Usage: MemTotal-MemAvailable in /proc/meminfo
    • Maximum value: MemTotal in /proc/meminfo
    ResolvConfFileProblem

    Check whether the ResolvConf file is lost.

    Check whether the ResolvConf file is normal.

    Exception definition: No upstream domain name resolution server (nameserver) is included.

    		 Object: /etc/resolv.conf
    ProcessD Check whether there is a process D on the node.

    Source:

    • /proc/{PID}/stat
    • Alternately, you can run ps aux.

    Exception scenario: ProcessD ignores the resident processes (heartbeat and update) that are in the D state that the SDI driver on the BMS node depends on.
    ProcessZ Check whether the node has processes in Z state.
    ScheduledEvent

    Check whether host plan events exist on the node.

    Typical scenario: The host is faulty, for example, the fan is damaged or the disk has bad sectors. As a result, cold and live migration is triggered for VMs.

    Source:

    This check item is an Alpha feature and is disabled by default.
    Table 3 Network connection check items
    Check Item Function Description
    CNIProblem Check whether the CNI component is running properly. None
    KUBEPROXYProblem Check whether kube-proxy is running properly. None
    Table 4 Storage check items
    Check Item Function Description
    ReadonlyFilesystem

    Check whether the Remount root filesystem read-only error occurs in the system kernel by listening to the kernel logs.

    Typical scenario: A user detaches a data disk from a node by mistake on the ECS, and applications continuously write data to the mount point of the data disk. As a result, an I/O error occurs in the kernel and the disk is reattached as a read-only disk.

    Listening object: /dev/kmsg

    Matching rule: Remounting filesystem read-only
    DiskReadonly Check whether the system disk, Docker disk, and kubelet disk are read-only.

    Detection paths:

    • /mnt/paas/kubernetes/kubelet/
    • /var/lib/docker/
    • /var/lib/containerd/
    • /var/paas/sys/log/cceaddon-npd/

    The temporary file npd-disk-write-ping is generated in the detection path.

    Currently, additional data disks are not supported.
    DiskProblem Check the usage of the system disk, Docker disk, and kubelet disk.

    • Threshold: 80%

    • Source:

      df -h
    Currently, additional data disks are not supported.
    EmptyDirVolumeGroupStatusError

    Check whether the ephemeral volume group on the node is normal.

    Impact: The pod that depends on the storage pool cannot write data to the temporary volume. The temporary volume is remounted as a read-only file system by the kernel due to an I/O error.

    Typical scenario: When creating a node, a user configures two data disks as a temporary volume storage pool. The user deletes some data disks by mistake. As a result, the storage pool becomes abnormal.

    • Detection period: 60s

    • Source:

      vgs -o vg_name, vg_attr

    • Principle: Check whether the VG (storage pool) is in the P state. If yes, some PVs (data disks) are lost.

    • Joint scheduling: The scheduler can automatically identify an abnormal node and prevent pods that depend on the storage pool from being scheduled to the node.

    • Exception scenario: The npd add-on cannot detect the loss of all PVs (data disks), resulting in the loss of VGs (storage pools). In this case, kubelet automatically isolates the node, detects the loss of VGs (storage pools), and updates the corresponding resources in nodestatus.allocatable to 0. This prevents pods that depend on the storage pool from being scheduled to the node. The damage of a single PV cannot be detected. In this case, the ReadonlyFilesystem detection is abnormal.
    LocalPvVolumeGroupStatusError

    Check the PV group on the node.

    Impact: Pods that depend on the storage pool cannot write data to the persistent volume. The persistent volume is remounted as a read-only file system by the kernel due to an I/O error.

    Typical scenario: When creating a node, a user configures two data disks as a persistent volume storage pool. Some data disks are deleted by mistake.
    MountPointProblem

    Check the mount point on the node.

    Exception definition: You cannot access the mount point by running the cd command.

    Typical scenario: Network File System (NFS), for example, obsfs and s3fs is mounted to a node. When the connection is abnormal due to network or peer NFS server exceptions, all processes that access the mount point are suspended. For example, during a cluster upgrade, a kubelet is restarted, and all mount points are scanned. If the abnormal mount point is detected, the upgrade fails.

    Alternatively, you can run the following command:

    for dir in `df -h | grep -v "Mounted on" | awk "{print \\$NF}"`;do cd $dir; done && echo "ok"
    DiskHung

    Check whether I/O faults occur on the disk of the node.

    Definition of I/O faults: The system does not respond to disk I/O requests, and some processes are in the D state.

    Typical Scenario: Disks cannot respond due to abnormal OS hard disk drivers or severe faults on the underlying network.

    • Check object: all data disks

    • Source:

      /proc/diskstat

      Alternatively, you can run the following command:

      iostat -xmt 1

    • Threshold:

      • Average usage. The value of ioutil is greater than or equal to 0.99.
      • Average I/O queue length. avgqu-sz >=1
      • Average I/O transfer volume, iops (w/s) + ioth (wMB/s) < = 1

      Note

      In some OSs, no data changes during I/O. In this case, calculate the CPU I/O time usage. The value of iowait is greater than 0.8.
    DiskSlow

    Check whether slow I/O occurs on the disk of the node.

    Definition of slow I/O: The average response time exceeds the threshold.

    Typical scenario: EVS disks have slow I/Os due to network fluctuation.

    • Check object: all data disks

    • Source:

      /proc/diskstat

      Alternatively, you can run the following command:

      iostat -xmt 1

    • Threshold:

      Average I/O latency: await > = 5000 ms

    Note

    If I/O requests are not responded and the await data is not updated. In this case, this check item is invalid.

    The kubelet component has the following default check items, which have bugs or defects. You can fix them by upgrading the cluster or using npd.

    Table 5 Default kubelet check items
    Check Item Function Description
    PIDPressure Check whether PIDs are sufficient.
    • Interval: 10 seconds
    • Threshold: 90%
    • Defect: In community version 1.23.1 and earlier versions, this check item becomes invalid when over 65535 PIDs are used. For details, see issue 107107. In community version 1.24 and earlier versions, thread-max is not considered in this check item.
    MemoryPressure Check whether the allocable memory for the containers is sufficient.
    • Interval: 10 seconds
    • Threshold: Max. 100 MiB
    • Allocable = Total memory of a node - Reserved memory of a node
    • Defect: This check item checks only the memory consumed by containers, and does not consider that consumed by other elements on the node.
    DiskPressure Check the disk usage and inodes usage of the kubelet and Docker disks.

    Interval: 10 seconds

    Threshold: 90%

Node-problem-controller Fault Isolation

Note

Fault isolation is supported only by add-ons of 1.16.0 and later versions.

When installing the npd add-on, set npc.enable to true to deploy dual Node-problem-controller (NPC). You can deploy NPC as single-instance but such NPC does not ensure high availability.

By default, if multiple nodes become faulty, NPC adds taints to only one node. You can set npc.maxTaintedNode to increase the threshold. When the fault is rectified, NPC is not running and taints remain. You need to manually clear the taints or start NPC.

The open source NPD plug-in provides fault detection but not fault isolation. CCE enhances the node-problem-controller (NPC) based on the open source NPD. This component is implemented based on the Kubernetes node controller. For faults reported by NPD, NPC automatically adds taints to nodes for node fault isolation.

You can modify add-onnpc.customConditionToTaint according to the following table to configure fault isolation rules.

Table 6 Parameters
Parameter Description Default
npc.enable Whether to enable NPC true
npc.customCondtionToTaint Fault isolation rules See Table 7 <cce_10_0132__table147438134911>.
npc.customConditionToTaint[i] Fault isolation rule items N/A

npc.customConditionToTaint[i].

condition.status

 Fault status true

npc.customConditionToTaint[i].

condition.type

 Fault type N/A

npc.customConditionToTaint[i].

enable

 Whether to enable the fault isolation rule. false

npc.customConditionToTaint[i].

.taint.effect

Fault isolation effect

NoSchedule, PreferNoSchedule, or NoExecute

NoSchedule

Value options: NoSchedule, PreferNoSchedule, and NoExecute
npc. maxTaintedNode

Number of nodes in a cluster that can be tainted by NPC

The int format and percentage format are supported.

1

Values:

  • The value is in int format and ranges from 1 to infinity.
  • The value ranges from 1% to 100%, in percentage. The minimum value of this parameter multiplied by the number of cluster nodes is 1.
Npc.affinity Node affinity of the controller N/A
Table 7 Fault isolation rule configuration
Fault Fault Details Taint
DiskReadonly Disk read-only NoSchedule: No new pods allowed.
DiskProblem The disk space is insufficient, and key logical disks are detached. NoSchedule: No new pods allowed.
FrequentKubeletRestart kubelet restarts frequently. NoSchedule: No new pods allowed.
FrequentDockerRestart Docker restarts frequently. NoSchedule: No new pods allowed.
FrequentContainerdRestart containerd restarts frequently. NoSchedule: No new pods allowed.
KUBEPROXYProblem kube-proxy is abnormal. NoSchedule: No new pods allowed.
PIDProblem Insufficient PIDs NoSchedule: No new pods allowed.
FDProblem Insufficient file handles NoSchedule: No new pods allowed.
MemoryProblem Insufficient node memory NoSchedule: No new pods allowed.

Collecting Prometheus Metrics

The NPD daemon pod exposes Prometheus metric data on port 19901. By default, the NPD pod is added with the annotation metrics.alpha.kubernetes.io/custom-endpoints: '[{"api":"prometheus","path":"/metrics","port":"19901","names":""}]'. You can build a Prometheus collector to identify and obtain NPD metrics from http://{{NpdPodIP}}:{{NpdPodPort}}/metrics.

Note

If the npd add-on version is earlier than 1.16.5, the exposed port of Prometheus metrics is 20257.

Currently, the metric data includes problem_counter and problem_gauge, as shown below.

# HELP problem_counter Number of times a specific type of problem have occurred.
# TYPE problem_counter counter
problem_counter{reason="DockerHung"} 0
problem_counter{reason="DockerStart"} 0
problem_counter{reason="EmptyDirVolumeGroupStatusError"} 0
...
# HELP problem_gauge Whether a specific type of problem is affecting the node or not.
# TYPE problem_gauge gauge
problem_gauge{reason="CNIIsDown",type="CNIProblem"} 0
problem_gauge{reason="CNIIsUp",type="CNIProblem"} 0
problem_gauge{reason="CRIIsDown",type="CRIProblem"} 0
problem_gauge{reason="CRIIsUp",type="CRIProblem"} 0
..