Why does the kubernetes environment require that bridge NF call iptables be enabled?

This article excerpts from kubernetes Learning notes

background

Kubernetes Environment , Many times, the node kernel parameters are required to be enabled bridge-nf-call-iptables:

sysctl -w net.bridge.bridge-nf-call-iptables=1

Refer to official documentation Network Plugin Requirements

If it is not turned on or the parameter is turned off due to some operation , It may cause some strange network problems , It's very troublesome to investigate .

Why open it ？ This article will tell you in detail .

Bridge based container network

Kubernetes There are many implementations of cluster networks , A large part of it is used Linux bridge :

• Every Pod All of the network cards are veth equipment ,veth pair The other end of is connected to the bridge on the host computer .
• Because the bridge is a virtual layer 2 device , Of the same node Pod The communication between them goes through layer 2 forwarding , Cross node communication will pass through the host eth0.

Service Communication with nodes

Whether it's iptables still ipvs Forwarding mode ,Kubernetes Medium visit Service It's all going on DNAT, Will visit ClusterIP:Port Data packets of DNAT become Service One of the Endpoint (PodIP:Port), Then the kernel inserts the connection information into conntrack Table to record connections , When the destination side returns the package, the kernel starts from conntrack Tables match joins and reverse NAT, In this way, the original path returns to form a complete connection link :

however Linux A bridge is a virtual layer-2 forwarding device , and iptables conntrack It's on the third floor , So if you directly access the address in the same bridge , It's going to go straight to the second level , Not pass conntrack:

1. Pod visit Service, Purpose IP yes Cluster IP, Not the address in the bridge , Three layers of forwarding , Will be DNAT become PodIP:Port.
2. If DNAT Then it was forwarded to the same node Pod, Purpose Pod I found the purpose when I returned the package IP On the same bridge , Just go straight to the second level forwarding , There is no call conntrack, The result is that there is no original way to return the packet ( See the picture below ).

Because there is no way back , The communication between client and server is not in the same place " channel " On , I don't think it's in the same connection , So we can't communicate properly .

The common problem phenomenon is the occasional occurrence DNS Parse failure , When coredns On the node pod analysis dns when ,dns The request falls on the current node coredns pod Upper time , This problem may occur .

Turn on bridge-nf-call-iptables

If Kubernetes In the network link of the environment bridge May encounter the above Service Communication with nodes , and Kubernetes Many network implementations use bridge.

Enable bridge-nf-call-iptables This kernel parameter ( Set as 1), Express bridge The device also calls when forwarding on the second layer iptables Three levels of rules for configuration ( contain conntrack), So turning on this parameter can solve the above problem Service Communication with nodes , That's why Kubernetes Environment , Most of them require opening bridge-nf-call-iptables Why .