MySQL master-slave replication, separation and resolution

One 、 summary

1.1 Application background

• In enterprise applications , Mature businesses usually have a large amount of data
• Single MySQL In security 、 High availability and high concurrency can not meet the actual needs
• Configure multiple master-slave database servers to realize read-write separation

1.2 What is separation of reading and writing ？

• Read / write separation , The basic principle is to make the primary database more transactional 、 Change 、 Delete operation （INSERT、UPDATE、DELETE）, And processing from the database SELECT Query operation .
• Database replication is used to synchronize changes caused by transactional operations to the slave database in the cluster .

1.3 Why read write separation ？

• Because the database “ Write ”（ Write 10000 This piece of data may be 3 minute ） The operation is time-consuming .
• But the database “ read ”（ read 10000 A piece of data may only 5 Second ）.
• So read write separation , The solution is , Database writing , Affect the efficiency of the query .

1.4 When to separate reading and writing ？

• The database does not have to be read-write separated , If the program uses more databases , And less updates , If there are many queries, we will consider using . Using database master-slave synchronization , Then, the database pressure can be shared through read-write separation , Improve performance .

1.5 Master slave replication is separated from reading and writing

• In the actual production environment , Read and write to the database in the same database server , It can't meet the actual needs . Whether it's in security 、 High availability or high concurrency can not meet the actual needs .
• therefore , Synchronize data through master-slave replication , Then through read-write separation to improve the concurrent load capacity of the database .
• It's kind of like rsync, But the difference is rsync It is used to back up disk files mysql Master-slave replication is the replication of data in the database 、 Statement for backup .

1.6 mysql Supported replication types

（1）STATEMENT： Statement based replication

• Execute on server sql sentence , Execute the same statement on the slave server ,mysql Statement based replication is used by default .

（2）ROW： Line based replication

• Copy the changes , Instead of executing the command on the slave server .

（3）MIXED： Hybrid type of replication

• Statement based replication is used by default , Once it is found that statement based cannot be accurately copied , Will adopt row based replication .

Two 、 The working process of master-slave replication

（1） Before each transaction updates the data ,Master In binary log （Binary log） Record these changes . After writing the binary log ,Master Notify the storage engine to commit the transaction .
（2）Slave take Master Copy the log to it （Relay log）. First slave Start a worker thread （I/O）,I/O The thread is in Master Open a normal connection on , But start Binlog dump process.Binlog dump process from Master Read events in binary log of , If and keep up Master, It will sleep and wait Master Create new events ,I/O Threads write these events to the relay log .
（3）SQL slave thread（SQL From thread ） The last step of processing this thread ,SQL The thread reads events from the relay log , And replays the events and updates slave data , Make it relate to master The data are consistent , As long as the thread and I/O Threads are consistent , The relay log is usually located at OS In cache , So the overhead of relay log is very small .
The replication process has one and important limitation , That is, to copy in Slave It's serial , in other words Master The parallel update operation on cannot be in Slave Up parallel operation .

The user is in mysql Write to the primary server , The master server logs the writes to the binary log , From the server through I/O The thread reads the binary log and writes it to the relay log of the slave server , From the server SQL The thread reads events from the relay log , And play it back .

3、 ... and 、 MySQL Read / write separation

3.1 mysql The principle of separation of reading and writing

• Read write separation is to write only on the primary server , Read only from service ;
• The main database handles transactional queries , Processing from database select Inquire about ;
• Database replication is used to synchronize changes caused by transactional queries to slave databases in the cluster

3.2 common MySQL There are two kinds of separation between reading and writing

1、 Based on the internal implementation of program code

• In the code according to select、insert Route classification , This kind of method is also the most widely used method in production environment .
  The advantage is better performance , Because it is implemented in program code , There is no need to add additional equipment for hardware expenses ; The disadvantage is that it needs developers to implement , Operation and maintenance personnel have no way to start .
• However, not all applications are suitable for reading and writing separation in program code , Like some large and complex Java application , If the separation of reading and writing is realized in the program code, the code will be greatly changed .

2、 Based on the intermediate agent layer

The proxy is usually located between the client and the server , After receiving the client request, the proxy server forwards it to the back-end database through judgment , There are the following representative procedures ：

• MySQL-Proxy.MySQL-Proxy by MySQL Open source project , Through its own lua The script goes on SQL Judge .
• Atlas. It's from Qihoo. 360 Of Web Platform Department infrastructure team development and maintenance based on MySQL Data middle tier project of the protocol . It's in mysql-proxy 0.8.2 Based on version , It's optimized , Added some new features .360 For internal use Atlas Running mysql Business , It carries billions of read and write requests every day . Support things and stored procedures .
• Amoeba. Developed by Chen Siru , The author worked for Alibaba . The procedure consists of Java Language development , Alibaba uses it in the production environment . But it does not support transactions and stored procedures .

• Due to the use MySQL Proxy Need to write a lot of Lua Script , these Lua Scripts are not ready-made , You need to write it yourself , This is not familiar with MySQL Proxy Built in variables and MySQL Protocol It's very difficult for people .

• Amoeba Is a very easy to use , Highly portable software , Therefore, it is widely used in the agent layer of database in production environment .

Four 、MySQL Principle of master-slave replication and read-write separation

The client sends a read / write operation to Amoeba The server ,Amoeba The server sends the write operation to the primary server , The master server logs the writes to the binary log , There are two threads from the server , One I/O A thread SQL Threads , From the server I/O Thread will be Master Open a normal connection on ,Binlog dump The thread will start from Master Read events in binary log of ,I/O Threads write these events to the relay log , Relay logs run in the cache ,SQL Thread reads events in relay log , And replay it in your own database ,Amoeba Read from two slave servers , Read events synchronized from the server .

5、 ... and 、 Deploy master-slave synchronization

Experimental thinking

1、 Client access proxy server
2、 The proxy server writes to the master server
3、 The master server overwrites the additions and deletions into its own binary log
4、 The slave server synchronizes the binary log of the master server to its own relay log
5、 Replay the relay log from the server to the database
6、 Client read , Then the proxy server directly accesses the slave server
7、 Reduce the load , Play a load balancing role

Environment configuration

host	operating system	IP Address	The tools needed / Software / Installation package
Master	CentOS7	192.168.61.11	mysql-5.7.20
Slave1	CentOS7	192.168.61.22	mysql-5.7.20
Slave2	CentOS7	192.168.61.44	mysql-5.7.20
client	CentOS7	192.168.61.55	----------

systemctl stop firewalld
systemctl disable firewalld
setenforce 0

5.1 Set up time synchronization （ master server ：192.168.61.11）

1.# Install the time synchronization server 
yum install ntp -y
 
2.# Modify the configuration file 
vim /etc/ntp.conf  
server 127.127.61.0   # Set the local clock source 
fudge 127.127.61.0 stratum 8  # Set the time level to 8  Restriction on 15  within 
 
3.# Opening service 
service ntpd start

1. Install the time synchronization server

2. Modify the configuration file

3. Opening service

5.2 Set up time synchronization （ From the server ：192.168.61.22、192.168.61.44）

1.# Install the time synchronization server 、 Synchronization service 
yum install ntp -y
yum install ntpdate -y
 
2. # Opening service 
service ntpd start
 
3. # Perform synchronization 
/usr/sbin/ntpdate 192.168.61.11
 
4.# Schedule scheduled tasks 
crontab -e
*/30 * * * *  /usr/sbin/ntpdate 192.168.61.11
 
###########slave2:192.168.61.44 Same as above ######

1. Install the time synchronization server 、 Synchronization service


2. Opening service

3. Perform synchronization

4. Schedule scheduled tasks

5.3 Configure the primary server （192.168.61.11）

1. # Turn on binary log 
vim /etc/my.cnf
 
log-bin=master-bin        # Turn on binary log 
binlog_format=MIXED       # Binary log format 
log-slave-updates=true    # Turn on synchronization from the server 
 
2. # Restart the service 
systemctl restart mysqld.service 
 
3. # Log in mysql, Authorize the slave server in the network segment 
mysql -uroot -p123456
grant replication slave on *.* to 'myslave'@'192.168.61.%' identified by '123456';
 
# Refresh database 
flush privileges;
 
# View the master server binaries 
show master status;
 Turn on binary log 

1 . Turn on binary log

2. Restart the service

3. Log in mysql, Authorize the slave server in the network segment

5.4 Configure slave （192.168.61.22、192.168.61.44）

1.# Turn on binary log 
vim /etc/my.cnf
server-id = 11 #slave1 and slave2 Of id Cannot be the same , I slave2 Set up 22
relay-log=relay-log-bin						# add to , Turn on relay log , Synchronize log records from the primary server to the local 
relay-log-index=slave-relay-bin.index		# add to , Define the location and name of the relay log file 
 
2.# Restart the service 
systemctl restart mysqld.service 
 
3. # Log in mysql, Configure synchronization note master_log_file and master_log_pos The value of and master Consistency of query 
mysql -uroot -p123123
change master to master_host='192.168.61.11',master_user='myslave',master_password='123456',master_log_file='master-bin.000001',master_log_pos=458;
 
4.# Start the synchronization , If you make a mistake , perform restart slave try 
start slave;
show slave status\G;
## The following two must be YES
#Slave_IO_Running: Yes
#Slave_SQL_Running: Yes
 
 
#########slave2：192.168.61.44 Same as above ######

1.# Turn on binary log

2.# Restart the service

3. # Log in mysql, Configure synchronization note master_log_file and master_log_pos The value of and master Consistency of query
   
   

5.5 Verify master-slave synchronization

# Create a library on the master server 
create database test_1;
 
# View... From the server 
show databases;