[tke] enable CPU static management strategy

background

By default , nodes Pod Default shared node CPU All in the pool CPU Check the number , When a lot of nodes are running CPU Dense Pod when , Workloads may switch schedules to different CPU nucleus , As a result, the performance of some workloads is significantly affected CPU The impact of cache affinity and scheduling latency . Regarding this ,kubelet Optional CPU Management strategy , Some key Pod Static binding of , avoid CPU The performance loss caused by preemption and switching to the business . Details refer to ： On the control node CPU Management strategy .

TKE How to enable configuration

The following is about TKE In the open CPU There are two ways to statically manage policies .

One 、 The stock node is enabled

1. static state CPU Policy configuration

stay K8S 1.17 Before the release , Available Pod Monopoly CPU The number of resources is equal to the number of nodes CPU Total minus passing --kube-reserved or --system-reserved Parameter reserved CPU, from 1.17 Version start ,CPU The reservation list can be accessed through kublet Of '--reserved-cpus' Parameters are set explicitly . because TKE Of GA edition Usually even numbers , therefore ：

• When TKE The cluster version is less than 1.18 when :

At the node /etc/kubernetes/kubelet Add the following configuration to the file ：

...
CPU_MANAGER_POLICY="--cpu-manager-policy=static"
KUBE_RESERVED="--kube-reserved=cpu=xxx,memory=xxx"
SYSTEM_RESERVED="--system-reserved=cpu=xxx,memory=xxx"
...

among , to Pods Monopoly CPU The number of resources available is equal to the number of nodes CPU Total minus passing --kube-reserved or --system-reserved Parameter reserved CPU, If you keep CPU If the quantity setting is not an integer, it will be rounded up , such as 250m, Rounding up is to keep 1 nucleus .

because TKE Of systemd Startup parameter environment variables are hard coded , So here we need to add kubelet Start parameter environment variable of CPU_MANAGER_POLICY and SYSTEM_RESERVED（KUBE_RESERVED The variable already exists by default ）：

modify kubelet Of systemd Startup file /usr/lib/systemd/system/kubelet.service, Add the following environment variables to the startup parameters ：

...
ExecStart=/usr/bin/kubelet ... ${CPU_MANAGER_POLICY} ${KUBE_RESERVED} ${SYSTEM_RESERVED} ...
...

• When TKE The cluster version is greater than or equal to 1.18 :

At the node /etc/kubernetes/kubelet Add the following configuration to the file ：

...
CPU_MANAGER_POLICY="--cpu-manager-policy=static"
RESERVED_CPUS="--reserved-cpus=xxx"
...

--reserved-cpus The parameter is explicitly specified as Pod Exclusive group CPU pool , Such as "--reserved-cpus=0,1,2,3", or CPU List of ranges "--reserved-cpus=0-3" It's all right .

Also modify kubelet Of systemd Startup file /usr/lib/systemd/system/kubelet.service, Add the following environment variables to the startup parameters ：

...
ExecStart=/usr/bin/kubelet ... ${CPU_MANAGER_POLICY} ${RESERVED_CPUS} ...
...

2. The policy configuration takes effect

Next, evict the nodes Pods, And delete kubelet Status files in the root directory cpu_manager_state To manually reset CPU Manager ：

1. stay TKE Console 【 The node list 】-> 【 more 】 spot 【 deportation 】 Option or execute the eviction command manually .
2. stay TKE Node cpu_manager_state File and restart kubelet service ：

rm /var/lib/kubelet/cpu_manager_state
systemctl  daemon-reload
systemctl  restart kubelet

Two 、 The newly added node is enabled （ recommend ）

Directly through customization kubelet Parameter to complete the above parameter addition , The policy configuration logic and the above inventory nodes （ First step ） The description is consistent , But at present, the user-defined parameters need to be supported , You can contact after-sales students to help open it .

summary

The above content only describes how to TKE Open static... In CPU Management strategy , About workload CPU How to configure the static binding core and precautions , see also static Strategy .