How to bind EIP to access public network in tke cluster fixed IP mode pod

Business deployment to tke In the cluster , Many times, third-party interface services need to be called through the public network , normal pod Access to the public network depends on the ability of nodes to access the public network , When a node accesses the public network, it can bind the public network to the node ip Or to vpc binding nat gateway . Actually in tke in , We can also give pod Binding elastic public network ip（eip）, Give Way pod By binding eip Go to the public network . Now let's talk about how to give pod To configure eip, And pass eip Go to the public network .

1. Fix ip Of pod binding eip

tke Intra cluster feeding pod binding eip, The premise is that pod The network mode of must be vpc-cni Fixation ip Pattern , If you create a cluster, choose GlobalRouter, Then start the cluster vpc-cni Mode is enough , If you create a cluster vpc-cni Pattern , It must be noted that the upper fixed ip,vpc-cni In mode , The cluster network mode cannot be changed after it is selected . When your cluster is fixed ip After the model , Refer to the examples in the documentation yaml to pod binding eip, I won't describe it in detail here .

https://cloud.tencent.com/document/product/457/50358

2. to update ip-masq-agent Version and configuration

Refer to the deployment in the previous step pod after ,pod You can't go straight eip Visit the public network , Why? ？ Because in tke There is a component in the cluster ip-masq-agent, This component will put the container network segment snat Go to the main network card to access the public network , therefore pod Access the public network or go out through the main network card , The public network accessed by the primary network card may be the public network of the primary network card ip perhaps nat Network segment , So now pod Access the public network or go to the public network of the node's primary network card ip perhaps nat The gateway eip.

To solve this problem ,tke This way ip-masq-agent Optimized and upgraded , Added a configuration NonMasqueradeSrcCIDRs, This configuration is the source of which configurations ip Do not do snat, In fact, it is in this configuration ip paragraph , Can't snat Become a node master network card ip, But go pod The secondary network card is out .

because NonMasqueradeSrcCIDRs This function has not been officially launched yet , The default version does not include this feature , If you want to use this function, you can update the cluster ip-masq-agent Component image version .

# kubectl edit ds -n kube-system ip-masq-agent -o yaml
 to update  spec.template.spec.containers[0].image tag  by  v2.6.0-alpha.0

After updating the component version , We also need to bind eip Of pod ip Add to ip-masq-agent-config Of NonMasqueradeSrcCIDRs Field , This means that you need to get what you need to add in advance pod ip, So we need to create a binding eip Of pod after , then pod ip Configuration to ip-masq-agent-config This configmap Inside , After configuration ,ip-masq-agent every other 1 Minutes will automatically load the configuration and take effect , Of course, it can also be rebuilt manually pod take effect .

# kubectl edit cm -n kube-system ip-masq-agent-config -o yaml

 The configuration mode is as follows 
apiVersion: v1
data:
  config: '{"NonMasqueradeCIDRs":["172.25.0.0/16","10.0.0.0/16"],"MasqLinkLocal":true,"ResyncInterval":"1m0s","NonMasqueradeSrcCIDRs":["10.0.33.2/32","10.0.33.3/32"]}'

3. modify vpc Routing table

After the second step is configured , If your node goes out to the public network, it is the node public network ip, No binding nat In the case of gateway ,pod You can directly use the binding eip I visited the public network , It doesn't need to be modified vpc Routing table .

But when your cluster node has no public network ip, The node is through nat The gateway accesses the public network , Or your node has a public network ip It also has nat gateway , So at this point pod Access to the public network still won't go eip, Or default to go nat gateway . Because in vpc In the routing policy of , When a subnet is associated with NAT gateway , And the ECS in the subnet has a public network IP（ Or elasticity IP） when , Will pass by default NAT Gateway access Internet（ Because the most accurate route has a higher priority than the public network IP）.

If you want to pod By binding eip Out of the public network , You need to add a policy ECS public network to the routing table IP visit Internet, such pod Only when you access the public network will you be bound eip.

4. test pod Visit the public network

When all the above changes are completed , We can simply test , Is it bound eip Of pod Going out of the public network is your own eip, Unbound eip Or go to the public network of the node's primary network card ip perhaps nat gateway （vpc The binding nat gateway ）

• Configure according to the following steps , The binding eip Of pod, The public network is bound here eip .

The test here meets the expectation .

• The node has a public network ip And there are also vpc binding nat gateway , But the routing table is configured with the ECS public network IP visit Internet, So the test is not bound eip Of pod, Out of the public network ip It is the public network of the node's primary network card ip.

The test here meets the expectation .

5. How to delete pod Fixation ip and eip

Fix ip The mode supports configuration pod Fix ip and eip Recycling strategy for . Respectively in yaml To configure tke.cloud.tencent.com/vpc-ip-claim-delete-policy

and tke.cloud.tencent.com/eip-claim-delete-policy this 2 A field , If not configured, the default is Immediate Recycle now , That means you destroy pod, Fix ip and eip It's all recycled , Fix ip and eip Whether to recycle , You can go to the cluster to view vipc and eipc this 2 Delete resource objects .

[[email protected] ~]# kubectl api-resources | grep ipc
eipclaims                                  eipc               networking.tke.cloud.tencent.com   true         EIPClaim
vpcipclaims                                vipc               networking.tke.cloud.tencent.com   true         VpcIPClaim

Example yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
 labels:
   k8s-app: busybox
 name: busybox
 namespace: default
spec:
 replicas: 3
 selector:
   matchLabels:
     k8s-app: busybox
     qcloud-app: busybox
 serviceName: ""
 template:
   metadata:
     annotations:
       tke.cloud.tencent.com/networks: "tke-route-eni"
       tke.cloud.tencent.com/vpc-ip-claim-delete-policy: Never
       tke.cloud.tencent.com/eip-attributes: ""
       tke.cloud.tencent.com/eip-claim-delete-policy: "Never"
     creationTimestamp: null
     labels:
       k8s-app: busybox
       qcloud-app: busybox
   spec:
     containers:
     - args:
       - "10000000000"
       command:
       - sleep
       image: busybox
       imagePullPolicy: Always
       name: busybox
       resources:
         limits:
           tke.cloud.tencent.com/eni-ip: "1"
           tke.cloud.tencent.com/eip: "1"
         requests:
           tke.cloud.tencent.com/eni-ip: "1"
           tke.cloud.tencent.com/eip: "1"

If you configure eip-claim-delete-policy and vpc-ip-claim-delete-policy Strategy is never, So you deleted pod, Corresponding eip And fixed ip Will not delete , You want to delete the fixed ip and eip Words , You can modify yaml, take tke.cloud.tencent.com/vpc-ip-claim-delete-policy and tke.cloud.tencent.com/eip-claim-delete-policy this 2 Delete comments , And then update pod that will do .

Of course, you can also modify yaml Remove fields , Then manually delete pod Corresponding eipc and vipc.