MySQL master-slave synchronization and its basic process of database and table division

Main memory synchronization

principle

mysql Three threads are designed in master-slave synchronization (log dump thread,i/o thread,sql thread), Two logs （bin log ,relay log）.

• Master node log dump Threads

  When the slave node connects to the master node , The master node creates a log dump Threads , Used for sending bin-log The content of . In the reading bin-log Operating time , This thread will change the bin-log Lock , When the read is complete , Even before launching to the slave node , The lock will be released .

• From the node I/O Threads When executed from a node start
  slave After the command , From the node, a I/O Threads are used to connect to the master node , Request the updated bin-log.I/O Thread received master binlog dump
  After the update from the process , Keep it locally relay-log in .

• From the node SQL Threads
  SQL The thread is responsible for reading relay log The content in , Parse into specific operations and execute , Finally, ensure the consistency of master-slave data .

Execute the process

After the database is updated, the updated data will be written to bin log among , From the slave node i/o The thread reads the master node bin log Update content to relay log in , Through sql The process is executing on the slave node .

Copy way

MySQL Master slave replication is asynchronous by default .MySQL All additions, deletions and modifications will be recorded in binary log in , When slave Node connection master when , From the active master Get the latest bin log file , And put bin log Medium sql adopt I/O relay To local .

• Asynchronous mode （mysql async-mode） In this mode , The master node will not take the initiative push bin
  log To the slave node （ After the master node completes execution, it submits directly ）, This could lead to failover Under the circumstances , Maybe the slave node did not update the latest bin log Sync to local .

• Full sync mode
  The full synchronization mode means that the master node and all the slave nodes execute commit And confirm to return success to the client
• Semi synchronous mode (mysql semi-sync)
  In this mode, the master node only needs to receive the return information from one of the slave nodes （ A slave node commit）, will commit; Otherwise, you need to wait until the timeout, and then switch to asynchronous mode to submit ;

binlog Record format

binlog There are three ways to generate

• be based on SQL Copy of sentences （statement-based replication,SBR）
• Line based replication （row-based replication,RBR)
• Hybrid mode replication （mixed-based replication,MBR)

be based on SQL Copy of sentences

Statement-base Replication (SBR) It's the record sql Statements in bin log in ,Mysql 5.1.4 And previous versions use this copy format .
advantage ： You only need to record the data that will be modified sql Statement to binlog in , Less binlog Daily quality , save I/O, Improve performance .
shortcoming ： In some cases , This will cause data inconsistency between master and slave nodes （ such as sleep(),now() etc. ）.

Line based replication

Row-based Relication(RBR) yes mysql master take SQL The statement is broken down to be based on Row Change the statement and record it in bin log in , That is, record which data has been modified , What's it going to look like .
advantage ： There will be no stored procedures under certain circumstances 、 Or functions 、 perhaps trigger Can't be copied correctly .
shortcoming ： There will be a lot of logs , Especially the revision table It's going to make the logs surge , At the same time increase bin log Synchronization time . It can't pass bin log Parse to get the executed sql sentence , You can only see what happened data change .

Hybrid mode replication

Mixed-format Replication(MBR),MySQL NDB cluster 7.3 and 7.4 The use of MBR. It's a mixture of the two modes , For general replication use STATEMENT Save the pattern to binlog, about STATEMENT If the mode cannot be copied, use ROW Pattern to save ,MySQL It will be executed according to SQL Statement to choose how to save the log .
Reference resources mysql Master-slave synchronization of - Principles

Sub database and sub table

Why do we need to divide the database and the table

There is too much data in the database , This leads to slower query efficiency , Previously used to increase the index , Read / write separation , When index optimization is not effective , To divide the database and table .
Single table bottleneck , Too much data , Query efficiency becomes low .
Single library bottleneck ,cpu, Memory , bandwidth , Disk pressure is too high .

Sub database and sub table mode

• Vertical sub table ： You can set the fields of a wide table according to the access frequency 、 Business coupling is loose and tight 、 The principle of whether a large field is split into multiple tables , This not only makes the business clear , It can also improve some performance . After break up , Try to avoid associated queries from a business perspective , Otherwise, the performance will not be worth it .
• Vertical sub database ： Multiple tables can be classified by business coupling , Store them in different warehouses , These libraries can be distributed on different servers , Thus, the access pressure is loaded by multiple servers , Greatly improve performance , At the same time, it can improve the business clarity of the overall architecture , Different business libraries can customize the optimization scheme according to their own situation . But it needs to solve all the complex problems brought about by cross Library .
• Horizontal sub database ： You can put the data of a table ( Press the data line ) It's divided into different warehouses , Each database has only part of the data of this table , These libraries can be distributed on different servers , Thus, the access pressure is loaded by multiple servers , Greatly improve performance . It not only needs to solve all the complex problems caused by cross Library , We also need to solve the problem of data routing .
• Horizontal sub table ： You can put the data of a table ( Press the data line ) It is divided into multiple tables in the same database , Each table has only part of the data of this table , This can slightly improve performance , It is only used as a supplementary optimization of the horizontal sub database .

Generally speaking , In the system design stage, we should determine the vertical sub base according to the business coupling tightness , Vertical table scheme , When the amount of data and access pressure is not particularly large , Consider caching first 、 Read / write separation 、 Index technology, etc . If the amount of data is huge , And continue to grow , Then consider the scheme of horizontal database and table .

Split steps

1. Objective assessment Split into several tables and several libraries
2. Segmentation strategy
   • Range segmentation Segment by field interval 1-100 101-200 Subsequent writes will not be evenly distributed
   • Intermediate table mapping Disadvantages the intermediate mapping table can be very large
   • hash segmentation （ recommend ） Data uniformity
3. Select the sub table field Minimize quark library queries
4. Resources to prepare
5. Double write New and old databases are written at the same time
6. Full data migration Put the old database data into the new database
7. Data consistency check
8. Grayscale cut reading Gradually migrate the traffic to the new database
9. Stop writing to the old library after cutting and reading .

Need tools

monitor bin Log Tools DataBus canal
Sub database and sub table tool Sharding-jdbc
Reference resources Billion level commodity data , Detailed explanation of the core process of sub database and sub table
Reason for sub warehouse and sub table , The way of sub database and sub table , Problems caused by sub database and sub table