vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/cce/umn/cce_bestpractice_10010.html
Dong, Qiu Jian e11d42fad0 CCE UMN update -20230818 version 
Reviewed-by: Eotvos, Oliver <oliver.eotvos@t-systems.com>
Co-authored-by: Dong, Qiu Jian <qiujiandong1@huawei.com>
Co-committed-by: Dong, Qiu Jian <qiujiandong1@huawei.com>
2023-12-08 10:20:34 +00:00

18 KiB
Raw Blame History

Pre-Binding Container ENI for CCE Turbo Clusters

In the Cloud Native Network 2.0 model, each pod is allocated an ENI or a sub-ENI (called container ENI). The speed of ENI creation and binding is slower than that of pod scaling, severely affecting the container startup speed in large-scale batch creation. Therefore, the Cloud Native Network 2.0 model provides the dynamic pre-binding of container ENIs to accelerate pod startup while improving IP resource utilization.

Constraints

  • CCE Turbo clusters of 1.19.16-r4, 1.21.7-r0, 1.23.5-r0, 1.25.1-r0 or later support ENI pre-binding, global configuration at the cluster level, and custom settings at the node pool level. Custom settings of nodes out of a node pools is not supported.
  • CCE Turbo clusters of 1.19.16-r2, 1.21.5-r0, 1.23.3-r0 to 1.19.16-r4, 1.21.7-r0, 1.23.5-r0 only support two parameters, nic-minimum-target and nic-warm-target, and do not support custom settings at the node pool level.
  • Modify the dynamic pre-binding parameters using the console or API instead of the node annotations in the background. Otherwise, the modified annotations will be overwritten by the original values after the cluster is upgraded.

How It Works

CCE Turbo provides four dynamic pre-binding parameters for container ENIs. You can properly configure the parameters based on your service requirements. (The node pool-level dynamic ENI pre-binding parameters take priority over the cluster-level dynamic ENI pre-binding parameters.)

Table 1 Parameters of the dynamic ENI pre-binding policy

Parameter

Default Value

Description

Suggestion

nic-minimum-target

10

Minimum number of ENIs bound to a node. The value can be a number or a percentage.

  • Value: The value must be a positive integer. For example, 10 indicates that at least 10 ENIs are bound to a node. If the ENI quota of a node is exceeded, the ENI quota is used.
  • Percentage: The value ranges from 1% to 100%. For example, 10%. If the ENI quota of a node is 128, at least 12 (rounded down) ENIs are bound to the node.

Set both nic-minimum-target and nic-maximum-target to the same value or percentage.

Set these parameters based on the number of pods.

nic-maximum-target

0

If the number of ENIs bound to a node exceeds the value of nic-maximum-target, the system does not proactively pre-bind NICs.

If the value of this parameter is greater than or equal to the value of nic-minimum-target, the check on the maximum number of the pre-bound ENIs is enabled. Otherwise, the check is disabled. The value can be a number or a percentage.

  • Value: The value must be a positive integer. For example, 0. The check on the maximum number of the pre-bound ENIs is disabled. If the ENI quota of a node is exceeded, the ENI quota is used.
  • Percentage: The value ranges from 1% to 100%. For example, 50%. If the ENI quota of a node is 128, the maximum number of the pre-bound ENI is 64 (rounded down).

Set both nic-minimum-target and nic-maximum-target to the same value or percentage.

Set these parameters based on the number of pods.

nic-warm-target

2

Extra ENIs will be pre-bound after the nic-minimum-target is used up in a pod. The value can only be a number.

When the value of nic-warm-target + the number of bound ENIs is greater than the value of nic-maximum-target, the system will pre-bind ENIs based on the difference between the value of nic-maximum-target and the number of bound ENIs.

Set this parameter to the number of pods that can be scaled out instantaneously within 10 seconds.

nic-max-above-warm-target

2

Only when the number of idle ENIs on a node minus the value of nic-warm-target is greater than the threshold, the pre-bound ENIs will be unbound and reclaimed. The value can only be a number.

  • Setting a larger value of this parameter slows down the recycling of idle ENIs and accelerates pod startup. However, the IP address usage decreases, especially when IP addresses are insufficient. Therefore, exercise caution when increasing the value of this parameter.
  • Setting a smaller value of this parameter accelerates the recycling of idle ENIs and improves the IP address usage. However, when a large number of pods increase instantaneously, the startup of some pods slows down.

Set this parameter based on the difference between the number of pods that are frequently scaled on most nodes within minutes and the number of pods that are instantly scaled out on most nodes within 10 seconds.

Configuration Example

Level

Service Scenario

Configuration Example

Cluster

All nodes use the c7.4xlarge.2 model (sub-ENI quota: 128).

Most nodes run about 20 pods.

Most nodes can run a maximum of 60 pods.

Most nodes can scale out 10 pods within 10 seconds.

Most nodes frequently scale in or out 15 pods within minutes.

Cluster-level global configuration:

  • nic-minimum-target: 20 or 16%
  • nic-maximum-target: 60 or 47%
  • nic-warm-target: 10
  • nic-max-above-warm-target: 5

Node pool

A node pool that uses the c7.8xlarge.2 high-specification model is created in the cluster. (sub-ENI quota: 256)

Most nodes run about 100 pods.

Most nodes can run a maximum of 128 pods.

Most nodes can scale out 10 pods within 10 seconds.

Most nodes frequently scale in or out 12 pods within minutes.

Custom settings at the node pool level:

  • nic-minimum-target: 100 or 40%
  • nic-maximum-target: 120 or 50%
  • nic-warm-target: 10
  • nic-max-above-warm-target: 2

Pods using HostNetwork are excluded.

Cluster-level Global Configuration

  1. Log in to the CCE console. In the navigation pane, choose Clusters.
  2. Click next to the target cluster.
  3. In the Manage Component window that is displayed on the sidebar, select Networking Components. For details about the parameter values, see Configuration Example.

  4. After the configuration is complete, click OK. Wait for about 10 seconds for the configuration to take effect.

Custom Settings at the Node Pool Level

  1. Log in to the CCE console.
  2. Click the cluster name to access the cluster console, choose Nodes on the left, and click the Node Pools tab on the right.
  3. Choose More > Manage next to the node pool name.
  4. In the Manage Component window that is displayed on the sidebar, select Networking Components. For details about the parameter values, see Configuration Example.

  5. After the configuration is complete, click OK. Wait for about 10 seconds for the configuration to take effect.
Parent topic: Networking