Cluster construction based on kubernetes v1.24.0 (II)

The last article mainly introduced , Environment configuration of each virtual machine . Now let's start with K8S Related deployment of .

In addition, I would like to add something from the previous article K8S Of changelog link ：
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md

1 To configure yum Source

All nodes need to do this

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

yum clean all

yum makecache

2 br_netfilter and ip_conntrack The module configuration

• All nodes need to do this

• load br_netfilter Module to enable the kernel ipv4 forward ,1.20+ Strongly dependent on this module

• ip_conntrack Module to record iptables The state of the network packet , And save every record to table in

modprobe br_netfilter
modprobe ip_conntrack

Here to prevent bridge-nf-call-ip6tables and bridge-nf-call-iptables Can't start , Need configuration br_netfilter Boot up automatically .

1. newly build /etc/rc.sysinit file

cat >>/etc/rc.sysinit<<EOF
#!/bin/bash
for file in /etc/sysconfig/modules/*.modules ; do
[ -x $file ] && $file
done
EOF

1. newly build br_netfilter.modules and ip_conntrack.modules, Re execution chmod 755 To increase permissions

echo "modprobe br_netfilter" >/etc/sysconfig/modules/br_netfilter.modules
echo "modprobe ip_conntrack" >/etc/sysconfig/modules/ip_conntrack.modules

3 Configure kernel forwarding and Bridge filtering

Add bridge filtering and kernel forwarding configuration files

cat <<EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF

net.bridge.bridge-nf-call-ip6tables： Whether in ip6tables Filtering in the chain IPv6 package
net.bridge.bridge-nf-call-iptables： Whether in iptables Filtering in the chain IPv4 package

Execute the following command , Make configuration effective

$ sysctl -p /etc/sysctl.d/k8s.conf

4 install ipset And ipvs

All hosts need to operate

4.1 install ipset

ipset yes iptables An extension of , It allows you to create rules that match the entire set of addresses . It's not like ordinary iptables The chain can only be single IP matching , ip The collection is stored in an indexed data structure , This kind of structure can be used for efficient search even if the real-time collection is large , Except for some common situations , For example, prevent some dangerous hosts from accessing this machine , So as to reduce system resource occupation or network congestion ,IPsets There are also some new firewall design methods , And simplify the configuration .

yum install ipset -y

4.2 install ipvs

IPVS (IP Virtual Server) Is based on Netfilter Of 、 As linux Part of the kernel implements the technology of load balancing on the transport layer .

IPVS Integrated into the LVS(Linux Virtual Server) in , It runs in the host , And act as a load balancer in front of a real server cluster .IPVS Can be right TCP/UDP The service's request is forwarded to the back-end real server , therefore IPVS Natural support Kubernetes Service.

With kubernetes Growth in usage , The scalability of its resources is becoming more and more important . Especially for the use of kubernetes For developers or companies running large workloads ,service Scalability of is critical .

kube-proxy Is for service Module for building routing rules , Previous dependence iptables To achieve the main service Type of support , such as (ClusterIP and NodePort). however iptables It is difficult to support tens of thousands of service, because iptables Designed purely for firewalls , And the underlying data structure is a list of kernel rules .

kubernetes As early as 1.6 Version already has the ability to support 5000 multi-node , This is based on iptables Of kube-proxy The cluster is expanded to 5000 Bottleneck of node . for instance , If in a 5000 Cluster of nodes , We created 2000 individual service, And each service Yes 10 individual pod, Then we will have at least on each node 20000 strip iptables The rules , This causes the kernel to be very busy .

be based on IPVS Load balancing in the cluster can solve this problem perfectly .IPVS It is specially designed for load balancing , And the bottom layer uses hash table, a very efficient data structure , Almost unlimited capacity expansion is allowed .

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

4.3 install ipvsadm

For ease of viewing ipvs The agency rules of , It's better to install the management tools ipvsadm.

yum install -y ipset ipvsadm

5 containerd

5.1 install containerd

containerd-{version}-linux-amd64.tar.gz Contains only containerd
cri-containerd-cni-{version}-linux-amd64.tar.gz It contains systemd The configuration file ,containerd as well as cni、runc And other related toolkits , Next we will use this package to install

open github containerd releases, View the latest packages , The latest version here is 1.6.6

But turn it on containerd.io, But the latest version is 1.6.4

I am downloading it later 1.6.6 At version time , Always report Connection refused

But download 1.6.4 When , But very smooth , I don't know why , But it doesn't affect our deployment , So let's just use 1.6.4.

wget https://github.com/containerd/containerd/releases/download/v1.6.4/cri-containerd-cni-1.6.4-linux-amd64.tar.gz

Then we unzip the package into the root directory of the system

tar zxvf cri-containerd-cni-1.6.4-linux-amd64.tar.gz -C /

5.2 To configure containerd

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

Set up the cgroup Drive as systemd

Replace default pause Mirror address

Set boot up

systemctl enable containerd --now

Test installation results

crictl version

Reference link

http://www.manongjc.com/detail/25-lioggelnywerjyf.html
https://www.kubernetes.org.cn/1904.html
https://www.toutiao.com/article/7105957860210819623/
https://i4t.com/5451.html
https://blog.frognew.com/2022/05/kubeadm-install-kubernetes-1.24.html
https://learn.lianglianglee.com/%E4%B8%93%E6%A0%8F/Kubernetes%20%E5%AE%9E%E8%B7%B5%E5%85%A5%E9%97%A8%E6%8C%87%E5%8D%97/07%20%E5%AE%B9%E5%99%A8%E5%BC%95%E6%93%8E%20containerd%20%E8%90%BD%E5%9C%B0%E5%AE%9E%E8%B7%B5.md