Configuring a Node Pool

Constraints

The default node pool DefaultPool does not support the following management operations.

Configuration Management

CCE allows you to highly customize Kubernetes parameter settings on core components in a cluster. For more information, see kubelet.

This function is supported only in clusters of v1.15 and later. It is not displayed for versions earlier than v1.15.

  1. Log in to the CCE console.
  2. Click the cluster name to access the cluster console. Choose Nodes in the navigation pane and click the Node Pools tab on the right.
  3. Click Manage in the Operation column of the target node pool
  4. On the Manage Components page on the right, change the values of Kubernetes parameters.

    Table 1 kubelet

    Item

    Parameter

    Description

    Value

    Modification

    CPU management policy

    cpu-manager-policy

    CPU management policy configuration. For details, see CPU Scheduling.

    • none: disables pods from exclusively occupying CPUs. Select this value if you want a large pool of shareable CPU cores.
    • static: enables pods to exclusively occupy CPUs. Select this value if your workload is sensitive to latency in CPU cache and scheduling.
    • enhanced-static: allows burstable pods to preferentially use CPU cores. Select this value if your workload has huge peak-trough difference and is in the trough state most of the time.

    Default: none

    None

    QPS for requests to kube-apiserver

    kube-api-qps

    Number of queries per second for communication with the API server.

    Default: 100

    None

    Burst for requests to kube-apiserver

    kube-api-burst

    Maximum number of burst requests sent to the API server per second.

    Default: 100

    None

    Limit on the pods managed by kubelet

    max-pods

    Maximum number of pods that can run on a node.

    None

    Limited number of processes in a pod

    pod-pids-limit

    Maximum number of PIDs that can be used in each pod.

    Default: -1, which indicates that the number of PIDs is not limited

    None

    Whether to use a local IP address as a node's ClusterDNS

    with-local-dns

    The default ENI IP address of the node will be automatically added to the node's kubelet configuration as the preferred DNS address.

    Default: false

    None

    QPS limit on creating events

    event-qps

    Number of events that can be generated per second.

    Default: 5

    None

    Allowed unsafe sysctls

    allowed-unsafe-sysctls

    Insecure system configuration allowed.

    Starting from v1.17.17, CCE enables pod security policies for kube-apiserver. Add corresponding configurations to allowedUnsafeSysctls of a pod security policy to make the policy take effect. (This configuration is not required for clusters earlier than v1.17.17.) For details, see Example of Enabling Unsafe Sysctls in Pod Security Policy.

    Default: []

    None

    Node oversubscription

    over-subscription-resource

    Whether to enable node oversubscription.

    If this parameter is set to true, node oversubscription is enabled on nodes. For details, see Dynamic Resource Oversubscription.
    • For clusters of versions earlier than v1.23.9-r0 or v1.25.4-r0: enabled (true) by default
    • Disabled by default if the cluster version is v1.23.9-r0, v1.25.4-r0, v1.27-r0, or later

    None

    Hybrid deployment

    colocation

    Whether to enable hybrid deployment on nodes.

    If this parameter is set to true, hybrid deployment is enabled on nodes. For details, see Dynamic Resource Oversubscription.
    • For clusters of versions earlier than v1.23.9-r0 or v1.25.4-r0: enabled (true) by default
    • Disabled by default if the cluster version is v1.23.9-r0, v1.25.4-r0, v1.27-r0, or later

    None

    Topology management policy

    topology-manager-policy

    Set the topology management policy.

    Valid values are as follows:

    • restricted: kubelet accepts only pods that achieve optimal NUMA alignment on the requested resources.
    • best-effort: kubelet preferentially selects pods that implement NUMA alignment on CPU and device resources.
    • none (default): The topology management policy is disabled.
    • single-numa-node: kubelet allows only pods that are aligned to the same NUMA node in terms of CPU and device resources.

    Default: none
    NOTICE:

    Modifying topology-manager-policy and topology-manager-scope will restart kubelet, and the resource allocation of pods will be recalculated based on the modified policy. In this case, running pods may restart or even fail to receive any resources.

    Topology management scope

    topology-manager-scope

    Configure the resource alignment granularity of the topology management policy. Valid values are as follows:

    • container (default)
    • pod

    Default: container

    Specified DNS configuration file

    resolv-conf

    DNS resolution configuration file specified by the container

    Default: null

    None

    Timeout for all runtime requests except long-running requests

    runtime-request-timeout

    Timeout interval of all runtime requests except long-running requests (pull, logs, exec, and attach).

    Default: 2m0s

    This parameter is available only in clusters of v1.21.10-r0, v1.23.8-r0, v1.25.3-r0, or later versions.

    Whether to allow kubelet to pull only one image at a time

    serialize-image-pulls

    Pull an image in serial mode.

    • false: recommended configuration so that an image can be pulled in parallel mode to improve pod startup.
    • true: allows images to be pulled in serial mode.
    • Enabled by default if the cluster version is earlier than v1.21.12-r0, v1.23.11-r0, v1.27.3-r0 or v1.25.6-r0
    • Disabled by default if the cluster version is v1.21.12-r0, v1.23.11-r0, v1.25.6-r0, v1.27.3-r0, or later

    This parameter is available only in clusters of v1.21.10-r0, v1.23.8-r0, v1.25.3-r0, or later versions.

    Image repository pull limit per second

    registry-pull-qps

    QPS upper limit of an image repository.

    Default: 5

    The value ranges from 1 to 50.

    This parameter is available only in clusters of v1.21.10-r0, v1.23.8-r0, v1.25.3-r0, or later versions.

    Upper limit of burst image pull

    registry-burst

    Maximum number of burst image pulls.

    Default: 10

    The value ranges from 1 to 100 and must be greater than or equal to the value of registry-pull-qps.

    This parameter is available only in clusters of v1.21.10-r0, v1.23.8-r0, v1.25.3-r0, or later versions.

    Node memory reservation

    system-reserved-mem

    System memory reservation reserves memory resources for OS system daemons such as sshd and udev.

    Default value: automatically calculated, which varies depending on node flavors. For details, see Node Resource Reservation Policy.

    The sum of kube-reserved-mem and system-reserved-mem must be less than 50% of the minimum memory of nodes in the node pool.

    kube-reserved-mem

    Kubernetes memory reservation reserves memory resources for Kubernetes daemons such kubelet and container runtime.

    Hard eviction

    memory.available

    Available memory on a node.

    The value is fixed at 100 MiB.

    For details, see Node-pressure Eviction.

    NOTICE:

    Exercise caution when modifying an eviction configuration item. Improper configuration may cause pods to be frequently evicted or fail to be evicted when the node is overloaded.

    kubelet can identify the following specific file system identifiers:

    • nodefs: main file system of a node. It is used for local disk volumes, emptyDir volumes that are not supported by memory, and log storage. For example, nodefs contains /var/lib/kubelet/.
    • imagefs: file system partition used by a container engine.

    nodefs.available

    Percentage of the available capacity in the filesystem used by kubelet.

    Default: 10%

    Value range: 1% to 99%

    nodefs.inodesFree

    Percentage of available inodes in the filesystem used by kubelet.

    Default: 5%

    Value range: 1% to 99%

    imagefs.available

    Percentage of the available capacity in the filesystem used by container runtimes to store resources such as images.

    Default: 10%

    Value range: 1% to 99%

    imagefs.inodesFree

    Percentage of available inodes in the filesystem used by container runtimes to store resources such as images.

    This parameter is left blank by default.

    Value range: 1% to 99%

    pid.available

    Percentage of allocatable PIDs reserved for pods.

    Default: 10%

    Value range: 1% to 99%

    Soft eviction

    memory.available

    Available memory on a node.

    The value must be greater than the hard eviction value of the same parameter, and the eviction grace period (evictionSoftGracePeriod) must be configured accordingly.

    This parameter is left blank by default.

    Value range: 100 to 1000000

    nodefs.available

    Percentage of the available capacity in the filesystem used by kubelet.

    The value must be greater than the hard eviction value of the same parameter, and the eviction grace period (evictionSoftGracePeriod) must be configured accordingly.

    This parameter is left blank by default.

    Value range: 1% to 99%

    nodefs.inodesFree

    Percentage of available inodes in the filesystem used by kubelet.

    The value must be greater than the hard eviction value of the same parameter, and the eviction grace period (evictionSoftGracePeriod) must be configured accordingly.

    This parameter is left blank by default.

    Value range: 1% to 99%

    imagefs.available

    Percentage of the available capacity in the filesystem used by container runtimes to store resources such as images.

    The value must be greater than the hard eviction value of the same parameter, and the eviction grace period (evictionSoftGracePeriod) must be configured accordingly.

    This parameter is left blank by default.

    Value range: 1% to 99%

    imagefs.inodesFree

    Percentage of available inodes in the filesystem used by container runtimes to store resources such as images.

    The value must be greater than the hard eviction value of the same parameter, and the eviction grace period (evictionSoftGracePeriod) must be configured accordingly.

    This parameter is left blank by default.

    Value range: 1% to 99%

    pid.available

    Percentage of allocatable PIDs reserved for pods.

    The value must be greater than the hard eviction value of the same parameter, and the eviction grace period (evictionSoftGracePeriod) must be configured accordingly.

    This parameter is left blank by default.

    Value range: 1% to 99%
    Table 2 kube-proxy

    Item

    Parameter

    Description

    Value

    Modification

    Maximum number of connection tracking entries

    conntrack-min

    Maximum number of connection tracking entries

    To obtain the value, run the following command:

    sysctl -w net.nf_conntrack_max

    Default: 131072

    None

    Wait time of a closed TCP connection

    conntrack-tcp-timeout-close-wait

    Wait time of a closed TCP connection

    To obtain the value, run the following command:

    sysctl -w net.netfilter.nf_conntrack_tcp_timeout_close_wait

    Default: 1h0m0s

    None
    Table 3 Network components (available only for CCE Turbo clusters)

    Item

    Parameter

    Description

    Value

    Modification

    Node pool ENI pre-binding

    enable-node-nic-configuration

    Whether to enable ENI pre-binding in a node pool.

    Default: false

    After network component configuration is disabled in a node pool, the dynamic container ENI pre-binding parameter settings of the node pool are the same as those of cluster-level parameter settings.

    ENI threshold

    nic-threshold

    Low threshold of the number of bound ENIs: High threshold of the number of bound ENIs

    Default: 0:0
    NOTE:

    This parameter is being discarded. Use the dynamic pre-binding parameters of the other four ENIs.

    Minimum number of ENIs bound to a node in a node pool

    nic-minimum-target

    Minimum number of container ENIs bound to a node.

    The parameter value must be a positive integer. The value 10 indicates that at least 10 container ENIs must be bound to a node. If the number you specified exceeds the container ENI quota of the node, the ENI quota will be used.

    Default: 10

    Configure these parameters based on the number of pods typically running on most nodes.

    Maximum number of ENIs pre-bound to a node in a node pool

    nic-maximum-target

    After the number of ENIs bound to a node exceeds the nic-maximum-target value, CCE will not proactively pre-bind ENIs.

    Checking the upper limit of pre-bound container ENIs is enabled only when the value of this parameter is greater than or equal to the minimum number of container ENIs (nic-minimum-target) bound to a node.

    The parameter value must be a positive integer. The value 0 indicates that checking the upper limit of pre-bound container ENIs is disabled. If the number you specified exceeds the container ENI quota of the node, the ENI quota will be used.

    Default: 0

    Configure these parameters based on the maximum number of pods running on most nodes.

    Number of ENIs dynamically pre-bound to a node in a node pool

    nic-warm-target

    Extra ENIs will be pre-bound after the nic-minimum-target is used up in a pod. The value can only be a number.

    When the sum of the nic-warm-target value and the current number of ENIs bound to the node is greater than the nic-maximum-target value, CCE will pre-bind on the number of ENIs specified by the difference between the nic-maximum-target value and the current number of ENIs bound to the node.

    Default: 2

    Set the parameter value to the number of pods that can be scaled out instantaneously within 10 seconds on most nodes.

    Threshold for reclaiming the ENIs pre-bound to a node in a node pool

    nic-max-above-warm-target

    Only when the difference between the number of idle ENIs on a node and the nic-warm-target value is greater than the threshold, the pre-bound ENIs will be unbound and reclaimed. The value can only be a number.

    • A large value will accelerate pod startup but slow down the unbinding of idle container ENIs and decrease the IP address usage. Exercise caution when performing this operation.
    • A small value will speed up the unbinding of idle container ENIs and increase the IP address usage but will slow down pod startup, especially when a large number of pods increase instantaneously.

    Default: 2

    Set the parameter value to the difference between the number of pods that are frequently scaled on most nodes within minutes and the number of pods that are instantly scaled out on most nodes within 10 seconds.
    Table 4 Pod security group in a node pool (available only for CCE Turbo clusters)

    Item

    Parameter

    Description

    Value

    Modification

    Default security group used by pods in a node pool

    security_groups_for_nodepool

    You can enter the security group ID. If this parameter is not configured, the default security group of the cluster container network will be used, and a maximum of five security group IDs that are separated by semicolons (;) can be specified at a time.

    The priority of the security group is lower than that of the security group configured for SecurityGroups.

    None

    None
    Table 5 Docker (available only for node pools that use Docker)

    Item

    Parameter

    Description

    Value

    Modification

    Container umask

    native-umask

    The default value normal indicates that the umask value of the started container is 0022.

    Default: normal

    The parameter value cannot be changed.

    Available data space for a single container

    docker-base-size

    Maximum data space that can be used by each container.

    Default: 0

    The parameter value cannot be changed.

    Insecure image source address

    insecure-registry

    Whether an insecure image source address can be used.

    false

    The parameter value cannot be changed.

    Maximum size of a container core file

    limitcore

    Maximum size of a core file in a container. The unit is byte.

    If not specified, the value is infinity.

    Default: 5368709120

    None

    Limit on the number of handles in a container

    default-ulimit-nofile

    Maximum number of handles that can be used in a container.

    Default: {soft}:{hard}

    The value cannot exceed the value of the kernel parameter nr_open and cannot be a negative number.

    You can run the following command to obtain the kernel parameter nr_open:

    sysctl -a | grep nr_open

    Image pull timeout

    image-pull-progress-timeout

    If the image fails to be pulled before time outs, the image pull will be canceled.

    Default: 1m0s

    This parameter is supported in v1.25.3-r0 and later.
    Table 6 containerd (available only for node pools that use containerd)

    Item

    Parameter

    Description

    Value

    Modification

    Available data space for a single container

    devmapper-base-size

    Maximum data space that can be used by each container.

    Default: 0

    The parameter value cannot be changed.

    Maximum size of a container core file

    limitcore

    Maximum size of a core file in a container. The unit is byte.

    If not specified, the value is infinity.

    Default: 5368709120

    None

    Limit on the number of handles in a container

    default-ulimit-nofile

    Maximum number of handles that can be used in a container.

    Default: 1048576

    The value cannot exceed the value of the kernel parameter nr_open and cannot be a negative number.

    You can run the following command to obtain the kernel parameter nr_open:

    sysctl -a | grep nr_open

    Image pull timeout

    image-pull-progress-timeout

    If the image fails to be pulled before time outs, the image pull will be canceled.

    Default: 1m0s

    This parameter is supported in v1.25.3-r0 and later.

    Verification on insure skips

    insecure_skip_verify

    Whether to skip repository certificate verification.

    Default: false

    The parameter value cannot be changed.

  5. Click OK.
Parent topic: Managing a Node Pool