当前位置：网站首页>LVS Technology Practice

LVS Technology Practice

2022-06-22 07:55:00 【yolo2016】

LVS yes Linux Virtual Server Abbreviation , meaning Linux Virtual server , Is a virtual server cluster system . This project is in 1998 year 5 Founded by Dr. Zhang wensong , It is one of the earliest free software projects in China .

LVS The cluster using IP Load balancing and content-based request distribution . The scheduler has a good throughput , Requests are evenly transferred to different servers for execution , And the scheduler automatically shields the server from failure , Thus, a set of servers is composed of a high-performance 、 Highly available virtual server . The structure of the entire server cluster is transparent to the customer , And there is no need to modify the client and server side of the program . So , The transparency of the system needs to be considered in the design 、 Scalability 、 High availability and manageability .

1,Cluster theory

1.1, Ways of system expansion ：

scale up： Upward development ;

scaleout： Outward development ;

1.2, Cluster type ：

LB: Load balancing cluster --> The loads faced by a system load Banlancing On another host ;
Hardware ： F5 Big-ip
Citrix Netscaler
A10 A10

Software ：Lvs（ Big websites ） There are no other health detection states except load balancing ;
haproxy, nginx ( Mostly ),httpd(proxy modular ) ,varnish
ats （apache traffic server） Development needs ;
perlbal

Based on the work protocol layer ：

Transport layer ： Lvs,haproxy( Analog out TCP Layer schedulable mode tcp);

application layer ：haproxy( Individual agreements ),nginx,ats,perlbal

HA: High availability cluster ,high availability

HP: High performance cluster ,high performancing

Distributed storage and parallel processing cluster ,

1.3, System ：

Extensibility ;

Usability ;

Capacity ; For example, highway , How many cars can you run ; The total amount completed per unit time ;

performance ; For example, speed , Number of cars passing per unit time ; response time ;

Be careful , The system operational ： Usability –> Do a good job in standardization –> Automatic operation and maintenance ( Stability above all else );
#################################################################################

2, Important principles for building high availability and scalability systems , Try to avoid walkthrough and interaction within the system ;
GSLB: Global serviceload balancing Global load balancing
SLB：service load balancing Service load balancing

layered ： Load balancing layer static content layer Data storage layer
Dynamic content layer

Division ： Functionally split , Distribute different requests to different server;
Distributed ：
distributed application ;
Distributed static resources ;
Distributed data and storage ;
Distributed computing ;

2.1, Four layer exchange , Layer 4 routing

According to the target of the request message IP and port Forward it to a host in the backend host cluster ( According to the load balancing algorithm );
netfilter：
Inbound ： prerouting------>input;
forward :prerouting—> forward—>postrouting;
Departure ：output---->postrouting;

2.2,Lvs ：

ipvsadm ( Write the rules )/ipvs( Define the rules ) Working in netfilter In the kernel of ;
ipvsadm： Command line tools in user space , Used to manage cluster services ;
ipvs： Working in the kernel netfilter INPUT Hook agreement ;
Support TCP、UDP、AH、EST、AH_EST、SCTP Such agreement ;

2.3,Centos 7 View the scheduling module information ;

grep -i -A 10 ‘IPVS’/boot/config-3.10.0-229.el7.x86_64

2.4,Lvs arch ：
Scheduler ：director( The director )、dispatcher( dispatcher )、
RS：Real server,
client IP：cip
directorvirutal ip：vip
director ip：dip
real serverip：rip

2.5,Lvs type：
A, Lvs-nat： be based on Dnat Pattern masquerade( Address camouflage ) type
B,Lvs-dr： Direct routing directrouting,gateway type
C,Lvs-tun：ip Tunnel ip tunnelingIPIP Tunnel type
D,Lvs-fullnat： Non standard type Modify the source when requesting message forwarding 、 Destination address

A,Lvs-nat：

Multi objective DNAT（iptables）;

By modifying the target of the request message IP Address and port , Complete port mapping , To pick out a RS Of RIP Address forwarding ;

（1） RS Should be and DIP Use a private address , And RS The gateway needs to point to DIP;

（2） Request and response messages are sent via director forward ; In a very high load scenario ,director It could be a system bottleneck ;

（3）RS You can use any OS, Support for port mapping ;

（4）RS Of RIP and director Of DIP Must be in the same IP The Internet ;

Pictured 1.1 Shown ：

B,Lvs-dr:direct routing

By modifying the target of the request message mac Address forwarding ;
director：VIP ,DIP
RSs：RIP,VIP

（1） It must be ensured that the front router will target IP by VIP The request message is sent to director;

Solution ：

Static binding ;director MAC Address binding

arptables; At every real server Enable arptables,

modify RS Host kernel parameters (linux host );

（2）RS Of RIP Private addresses can be used , But you can also use a public address ;

（3）RS root director Must be on the same physical network ;

（4） Request message via director Dispatch , However, the response message may not pass through director（ from RS Forward directly ）;

（5） Port mapping is not supported

（6）RS Yes, most OS;

（7）RS The gateway of cannot point to DIP;

#################################################################################
3,Lvs -type
Lvs-nat：RIP And DIP Must be in the same network segment ;
Lvs-dir: director And RS Must be in the same network segment ;
Lvs-tun：

Do not modify the request message IP The first one , But through the original IP Outside the head , It's encapsulating a IP The first one ;

（1）RIP、DIP、VIP All public network addresses ;

（2）RS Gateway cannot point to DIP,

（3） Request message via director Dispatch , The response message must not be sent via director;

（4） Port mapping is not supported ,RS Of OS Tunnel function must be supported ;

Lvs-fullnat: It can be set across machine rooms

When the request message arrives ,director At the same time, modify the source address and destination address of the message ;

（1）VIP It's the public address ,DIP and RIP Can not be in the same network segment ;

（2）RS The source address of the received request message is DIP, So respond to DIP;

（3） Request and response messages must be sent via director;

（4） Support for port mapping ,RS Of OS You can use any OS;

4,http：stateless

Track every client , You need to issue one for each client cookie;

session keep ： Requests from the same user , Call to the same RS;

session binding ： Request message insertion cookie, (nginx Sure ,lvs Can not be

session colony ;session The master-slave mechanism wastes memory 、cpu Equal consumption ;

session The server ; Shared storage server ,KV Mechanism storage ;

5,Lvs scheduler：

Static methods ： Scheduling only according to the algorithm itself , Regardless of the load state ; The starting point is fair

RR: roundrobin , polling

WRR：weighted rr , weighting

SH：source hash source address hash Realization session keep ; Damage the effect of load balancing ;

Will come from the same IP Requests are always scheduled to the same RS; Reverse proxy

DH：destination hash, Destination address hash, No matter which request comes from , When requesting the same target , Will be dispatched to the same RS, Forward agency

Dynamic methods ： According to the algorithm and each RS The current load state of the ;sh
LC：least connection Minimum number of connections ;
overhead=active( Number of active links )*256+inactive( Number of inactive connections )
WLC：weighted least connection Weighted least connected ;
overhead=（active( Active )*256+inactive( Inactive )）/weight Server weight
SED： In the short term, we hope to delay shortest expection delay
overhead=（active+1）*256/weight
NQ：never queue
SED Algorithm improvements
LBLC：locality-based lc, dynamic DH Algorithm , When implementing a forward proxy ;
LBLCR： With duplicate LBLC
Significant response request , Allocation delay
#######################################################################################
static state
Locality ： Space + Time
When making cache , Memory based caching

Dynamic content ： Time + given
When making cache , Judge according to the user's time ;

######################################################################

6,Lvs Cluster services for ：TCP、UDP、ah、esp

（1） One ipvs The host can define multiple at the same time clusterserver;
according to tcp,udp
（2） One cluster server There's at least one real server;
Definition time ： To specify lvs-type as well as lvs scheduler（ Programming machine ）;

7,ipvsadm Usage of ：
A, Manage cluster services ;
ipvsadm-A|E -t|u|f service-address [-s scheduler]
Add and modify
ipvsadm-D -t|u|f service-address
Delete
ipvsadm-C Delete
ipvsadm -L | l [options]
service-address：
tcp:-t ip:port
udp:-u ip:port
fwm:-f mark Digital firewall flag ;
-sscheculer: The default is wlc Number of inactive Links

B, Manage... In cluster services RS;
ipvsadm -a|e -t|u|f service-address -rserver-address [-g|i|m] [-wweight]
ipvsadm -d -t|u|f service-address -r server-address

server-address;
ip [:port] Support for port mapping ; Most ports can be omitted ;
Lvs-type：
-g:gateway,dr
-i:ipip, tun
-m:masquerade,nat nat Model

C, Empty and view ：
ipvsadm -C
ipvsadm -L | l[options]
-n:numberic Display address and port based on digital format ;
-c:connection Show ipvs The number of connections ;
–stats: statistics ;pkts： The total number of packages
–rate： rate ; pps Number of packets per second
–exact： The exact value ;
–sort： Sort ;
D, Save and reload ：
ipvsadm -R heavy load
ipvsadm -S [-n] preservation
E, Zero counter ：
ipvsadm -Z[-t|u|f service-address]

######################################################################################################
Two , The practical part
Lvs-nat

1,DIP Set up ：director Prepare two network ports , One is the bridge , One is a virtual network card 2;

cd /etc/sysconfig/network-scripts/

vimifcfg-eno16777736

#ifconfig -a
see eno16777736 Address

Set another port vmnet2：

cpifcfg-eno16777736 ifcfg-eno33554984

vim ifcfg-eno33554984 Delete UUID change NAME、DEVICE Parameters ;

2, Next, we configure the other two web machine ：
Two sets are configured respectively web Server's IP Address , and web service ;
And then in director The above tests two pages ; And shut down iptables Protective wall ;
#curl http://192.168.20.7
#curl http://192.168.20.8
#iptables -F
stay director The up forwarding function is enabled

cat /proc/sys/net/ipv4/ip_forward

vim /etc/sysctl.conf Edit add "net.ipv4.ip_forword = 1"

3, Now in director Set rules above ：

ipvsadm -A -t 172.16.100.117:80 -s rr

ipvsadm -a -t 172.16.100.117:80 -r192.168.20.7 -m

ipvsadm -a -t 172.16.100.117:80 -r192.168.20.8 -m

4, Save the configuration permanently ：
]# ipvsadm -S > /etc/sysconfig/ipvsadm

5, If you accidentally delete a rule , It can be repaired in time ;

6, Reread profile , And repair ;

ipvsadm -R < /etc/sysconfig/ipvsadm

7, Start validation Lvs-nat effect ：
http://172.16.100.117 Check it out. ;

8, Use "-E" Option to modify the scheduler ：
#ipvsadm -E -t 172.16.100.117:80 -s sh

9, be based on web modify RS Port mapping ;

ipvsadm -e -t172.16.100.117:80 -r 192.168.20.7:8080 -m

ipvsadm -e -t172.16.100.117:80 -r 192.168.20.8:8080 -m

Or modify #vim/etc/sysconfig/ipvsadm In the parameters of the ; Then reread the message .
wKioL1Yfm3XyKQg_AAEPFaaq0qQ904.jpg

such web The service is back to normal ;

10, Delete one real server;
#ipvsadm -d -t 172.16.100.117:80 -r192.168.20.7:8080

Delete cluster service ：
#ipvsadm -D -t 172.16.100.117:80

Repair ipvsadm
#ipvsadm -R < /etc/sysconfig/ipvsadm

11,Lvs-dr Kernel parameter configuration arp_ignore Whether to respond to the request / arp_announce Whether to accept the request notice
arp_announce( notice )-INTEGER request
0 Default , Do not keep announcements
1 Try not to announce
2 Can only 、 Always avoid announcements
arp_announce = 1 Try to avoid ,
arp_announce = 2 Can only 、 Always avoid ,

arp_ignore -INTEGER Respond to
0 - Default Inform the audience
1 - Only notify the entry of the message request ,
Lvs-dr Operation chart ：

A, modify DIP
To configure director Host address and loopback address ：