Cluster 2 - LVS load balancing cluster Dr mode

1 Packet flow analysis

（1） Client sends request to Director Server（ Load Balancer ）, The requested datagram （ Source IP yes CIP, The goal is IP yes VIP） Get to kernel space .
（2）Director Server and Real Server In the same network , Data is transmitted through the layer 2 data link layer .
（3） Kernel space determines the destination of the packet IP It's local VIP, here IPVS（IP Virtual server ） Compare whether the service requested by the packet is a cluster service , If it's a cluster service, repackage the data package .
Modification source MAC The address is Director Server Of MAC Address , Modify target MAC The address is Real Server Of MAC Address , Source IP Address and destination IP The address hasn't changed , The packet is then sent to the Real Server.
（4） arrive Real Server Of the request message MAC The address is its own MAC Address , This message is received . Packet repackaging message （ Source IP The address is VIP, The goal is IP by CIP）, Pass the response message lo. Interface to the physical network card, and then sent out .
（5）Real Server Send the response message to the client directly .

2 DR Characteristics of the pattern

（1）Director Server and Real Server Must be in the same physical network .
（2）Real Server Private addresses can be used , You can also use a public address . If you use a public address , You can do it through the Internet RIP Make a direct visit .
（3）Director Server As an access portal to the cluster , But not as a gateway .
（4） All request messages are sent through Director Server, But the response message cannot pass through Director Server.
（5）Real Server The gateway of is not allowed to point to Director Server IP, namely Real Server Packets sent are not allowed to pass through Director Server.
（6）Real Server Upper lo Interface configuration VIP Of IP Address .
1、 Packet flow analysis
(1） Client sends request to Director Server( Load Balancer ）, The requested datagram （ Source IP yes CIP, The goal is IP yes VIP) Get to kernel space .

(2)Director server and Real Server In the same network , Data is transmitted through the layer 2 data link layer .
(3） Kernel space determines the destination of the packet IP It's local VIP, here IPVS（IP Virtual server ） Compare whether the service requested by the packet is a cluster service , If it's a cluster service, repackage the data package . Modification source MAc The address is Director Server Of MAC Address , Modify target MAC The address is Real Server Of MAC Address , Source IP Address and destination IP The address hasn't changed , The packet is then sent to the Real server.
(4) arrive Real Server Of the request message NAc The address is its own wAc Address , This message is received . Packet repackaging message ( Source ⅠP The address is VIP, The goal is IP by CIP), Pass the response message lo Interface to the physical network card, and then sent out .
(5) Real server Direct will l The response message is transmitted to the client .

3 LVS-DR Medium ARP problem

3.1 IP Address conflict

stay LVS-DR Load balancing cluster , Load balancer and node server should be configured the same VIP Address , Have the same... In a LAN IP Address . Have the same address in the LAN , It's bound to cause servers ARP Communication disorder

When ARP Broadcast to LVS-DR When the cluster , Because the load balancer and the node server are connected to the same network , They will all receive ARP radio broadcast
Only the front-end load balancer responds , Other node servers should not respond ARP radio broadcast
terms of settlement
Process the node server , Make it unresponsive VIP Of ARP request

Virtual interface lo:0 bearing VIP Address
Set kernel parameters arp_ ignore=1: The system only responds to the purpose IP For the local IP Of ARP request
After setting, the node server will not respond ARP radio broadcast , The scheduler will still respond ARP Therefore, it can only be resolved to the scheduler MAC Address

3.2 Routing is based on ARP Table item , Will forward the new request message to RealServer, Lead to Director Of VIP invalid

RealServer Return message （ Source IP yes VIP） Forward via router , When repacking a message , You need to get the router first MAC Address ,
send out ARP When asked ,Linux By default IP Source of package IP Address （ namely VIP） As ARP Request source in package IP Address , Instead of using the sending interface IP Address ,
Router received ARP After the request , Will be updated ARP Table item , The original VIP Corresponding Director Of MAC The address will be updated to VIP Corresponding RealServer Of MAC Address .
The router is based on ARP Table item , Will forward the new request message to RealServer, Lead to Director Of VIP invalid
resolvent
Process the node server

Set kernel parameters arp_announce=2
The system doesn't use IP Set the source address of the package ARP The source address of the request , And choose the send interface IP Address

3.3 How to set up two problems

modify /etc/sysctl.conf file

 Node server in lo：0 The virtual interface carries VIP Address 
net.ipv4.conf.lo.arp_ignore = 1		  # Make the local system respond only to the purpose IP For the local physical network card IP Of ARP request 
net.ipv4.conf.lo.arp_announce = 2     # The native system does not use the source address of the returned packet as the source address ARP The source address of the request message , And using the sending interface IP As ARP Request the source address 
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

4 Deploy DR Pattern

 Scheduling server 192.168.16.16
NFS The server 192.168.16.18
web1 The server 192.168.16.20
web2 The server 192.168.16.22
 client 192.168.16.24
VIP192.168.16.26

192168.16.16 Scheduling server

 Turn off the firewall and selinux, download ipvsadm
systemctl stop firewalld.service 
setenforce 0
yum install ipvsadm -y

Configure virtual IP Address

[[email protected] network-scripts]# ls
ifcfg-ens33  ifdown-ippp    ifdown-sit       ifup-bnep  ifup-plip    ifup-Team          network-functions-ipv6
ifcfg-lo     ifdown-ipv6    ifdown-Team      ifup-eth   ifup-plusb   ifup-TeamPort
ifdown       ifdown-isdn    ifdown-TeamPort  ifup-ib    ifup-post    ifup-tunnel
ifdown-bnep  ifdown-post    ifdown-tunnel    ifup-ippp  ifup-ppp     ifup-wireless
ifdown-eth   ifdown-ppp     ifup             ifup-ipv6  ifup-routes  init.ipv6-global
ifdown-ib    ifdown-routes  ifup-aliases     ifup-isdn  ifup-sit     network-functions
[[email protected] network-scripts]# cp ifcfg-ens33 ifcfg-ens33:0
[[email protected] network-scripts]# vim ifcfg-ens33:0
 
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.16.26
NETMASK=255.255.255.0
#GATEWAY=192.168.16.2
#DNS1=8.8.8.8

Restart NIC

[[email protected] network-scripts]# ifdown ifcfg-ens33:0
[[email protected] network-scripts]# ifup ifcfg-ens33:0

adjustment proc Response parameter

[[email protected] network-scripts]# vim /etc/sysctl.conf 
 
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

Refresh configuration

[[email protected] network-scripts]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

Load module

[[email protected] network-scripts]# modprobe ip_vs
[[email protected] network-scripts]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn

Configure load distribution policies , Start the service

[[email protected] network-scripts]# ipvsadm-save >/etc/sysconfig/ipvsadm
[[email protected] network-scripts]# systemctl start ipvsadm.service

Empty ipvsadm

[[email protected] network-scripts]# ipvsadm -C
 Add a policy 
[[email protected] network-scripts]# ipvsadm -A -t 192.168.16.26:80 -s rr
[[email protected] network-scripts]# ipvsadm -a -t 192.168.16.26:80 -r 192.168.16.20:80 -g
[[email protected] network-scripts]# ipvsadm -a -t 192.168.16.26:80 -r 192.168.16.22:80 -g

Save settings

[[email protected] network-scripts]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  localhost.localdomain:http rr
  -> 192.168.16.20:http          Route   1      0          0         
  -> 192.168.16.22:http          Route   1      0          0         
[[email protected] network-scripts]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.16.26:80 rr
  -> 192.168.16.20:80            Route   1      0          0         
  -> 192.168.16.22:80            Route   1      0          0         
[[email protected] network-scripts]# ipvsadm-save >/etc/sysconfig/ipvsadm

192.168.16.18 NFS The server

[[email protected] ~]# cd /opt/
[[email protected] opt]# ls
rh
[[email protected] opt]# mkdir nfs
[[email protected] opt]# cd nfs/
[[email protected] nfs]# mkdir my qyd
[[email protected] nfs]# ls
my  qyd
[[email protected] nfs]# echo "this is my" >my/index.html
[[email protected] nfs]# echo "this is qyd" >qyd/index.html

Set the permissions

[[email protected] nfs]# chmod 777 *
[[email protected] nfs]# ll
 Total usage  0
drwxrwxrwx. 2 root root 24 5 month   13 16:30 my
drwxrwxrwx. 2 root root 24 5 month   13 16:30 qyd

Set sharing policy

[[email protected] nfs]# vim /etc/exports
 
/opt/nfs/my 192.168.16.0/24(rw,sync,no_root_squash)
/opt/nfs/qyd 192.168.16.0/24(rw,sync,no_root_squash)

Open service publishing sharing

[[email protected] nfs]# systemctl start rpcbind
[[email protected] nfs]# systemctl start nfs
[[email protected] nfs]# showmount -e
Export list for localhost.localdomain:
/opt/nginx/qyd 192.168.16.0/24
/opt/nginx/my  192.168.16.0/24

192.168.16.20 web1 The server

 Turn off firewall 

[[email protected] ~]# systemctl stop firewalld.service 
[[email protected] ~]# setenforce 0

[[email protected] ~]# showmount -e 192.168.16.18
Export list for 192.168.16.18:
/opt/nfs/qyd 192.168.16.0/24
/opt/nfs/my  192.168.16.0/24

install apache service

[[email protected] ~]# yum install httpd -y
 mount 
[[email protected] ~]# mount 192.168.16.18:/opt/nfs/my /var/www/html/
[[email protected] ~]# df -h
 file system                      Capacity    Already used    You can use   Already used %  Mount point 
/dev/mapper/centos-root      20G  3.7G   17G   19% /
devtmpfs                    473M     0  473M    0% /dev
tmpfs                       489M     0  489M    0% /dev/shm
tmpfs                       489M  7.2M  481M    2% /run
tmpfs                       489M     0  489M    0% /sys/fs/cgroup
/dev/sda1                  1014M  161M  854M   16% /boot
tmpfs                        98M     0   98M    0% /run/user/0
tmpfs                        98M   12K   98M    1% /run/user/42
192.168.16.18:/opt/nfs/my   10G  3.7G  6.4G   37% /var/www/html
[[email protected] ~]# cd /var/www/html/
[[email protected] html]# ls
index.html
[[email protected] html]# cat index.html 
this is my

Restart the service and enter IP Address check

[[email protected] html]# systemctl restart httpd.service 

configure gateway

Restart NIC

[[email protected] network-scripts]# systemctl restart network
[[email protected] network-scripts]# ifconfig

Set the routing

[[email protected] network-scripts]# route add -host 192.168.16.26 dev lo:0
[[email protected] network-scripts]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.16.2    0.0.0.0         UG    100    0        0 ens33
192.168.16.0    0.0.0.0         255.255.255.0   U     100    0        0 ens33
192.168.16.26  0.0.0.0         255.255.255.255 UH    0      0        0 lo
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

Start up and execute the command

[[email protected] network-scripts]# vim /etc/rc.d/rc.local 

#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
 
/usr/sbin/route add -host 192.168.52.188 dev lo:0
[[email protected] network-scripts]# chmod +x /etc/rc.d/rc.local

adjustment proc Response parameter

[[email protected] network-scripts]# vim /etc/sysctl.conf

# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[[email protected] network-scripts]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2

192.168.16.22 web2 The server
Turn off firewall

[[email protected] ~]# systemctl stop firewalld.service 
[[email protected] ~]# setenforce 0


[[email protected] ~]# showmount -e 192.168.52.110
Export list for 192.168.16.18:
/opt/nfs/qyd 192.168.16.0/24
/opt/nfs/my  192.168.16.0/24

install apache service

[[email protected] ~]# yum install httpd -y
 mount 
[[email protected] ~]# mount 192.168.16.18:/opt/nfs/qyd /var/www/html/
[[email protected] ~]# df -h
 file system                       Capacity    Already used    You can use   Already used %  Mount point 
/dev/mapper/centos-root       15G  3.7G   12G   25% /
devtmpfs                     897M     0  897M    0% /dev
tmpfs                        912M     0  912M    0% /dev/shm
tmpfs                        912M  9.1M  903M    1% /run
tmpfs                        912M     0  912M    0% /sys/fs/cgroup
/dev/sda1                    497M  172M  326M   35% /boot
tmpfs                        183M  4.0K  183M    1% /run/user/42
tmpfs                        183M   44K  183M    1% /run/user/0
192.168.16.18:/opt/nfs/qyd   10G  3.7G  6.4G   37% /var/www/html
[[email protected] ~]# cd /var/www/html/
[[email protected] html]# ls
index.html
[[email protected] html]# cat index.html 
this is qyd

Restart the service and enter IP Address check

[[email protected] html]# systemctl restart httpd.service 

configure gateway

[[email protected] html]# cd /etc/sysconfig/network-scripts/
[[email protected] network-scripts]# ls
ifcfg-ens33  ifdown-ipv6    ifdown-TeamPort  ifup-ippp   ifup-routes       network-functions
ifcfg-lo     ifdown-isdn    ifdown-tunnel    ifup-ipv6   ifup-sit          network-functions-ipv6
ifdown       ifdown-post    ifup             ifup-isdn   ifup-Team
ifdown-bnep  ifdown-ppp     ifup-aliases     ifup-plip   ifup-TeamPort
ifdown-eth   ifdown-routes  ifup-bnep        ifup-plusb  ifup-tunnel
ifdown-ib    ifdown-sit     ifup-eth         ifup-post   ifup-wireless
ifdown-ippp  ifdown-Team    ifup-ib          ifup-ppp    init.ipv6-global
[[email protected] network-scripts]# cp ifcfg-lo ifcfg-lo:0
[[email protected] network-scripts]# vim ifcfg-lo:0
 
DEVICE=lo:0
IPADDR=192.168.16.26
NETMASK=255.255.255.255

Restart the service and see

[[email protected] network-scripts]# systemctl restart network
[[email protected] network-scripts]# ifconfig

Set the routing

[[email protected] network-scripts]# route add -host 192.168.16.26 dev lo:0
[[email protected] network-scripts]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.16.2    0.0.0.0         UG    100    0        0 ens33
192.168.16.0    0.0.0.0         255.255.255.0   U     100    0        0 ens33
192.168.16.26  0.0.0.0         255.255.255.255 UH    0      0        0 lo
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

Start up and execute the command

[[email protected] network-scripts]# vim /etc/rc.d/rc.local 
 
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
 
/usr/sbin/route add -host 192.168.16.26 dev lo:0
[[email protected] network-scripts]# chmod +x /etc/rc.d/rc.local

adjustment proc Response parameter

[[email protected] network-scripts]# vim /etc/sysctl.conf

# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[[email protected] network-scripts]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2

Client test 192.168.16.24