Usage and introduction of MySQL binlog

binlog

One 、 What is? binlog?

• binlog It's a binary file , Can be said to be MySQL The most important log , It records everything DDL(create、alter、drop) and DML sentence （ In addition to data query statements select）, Record as an event , It also contains the time consumed by statement execution ,MySQL The binary log of is transaction safe .
• By default ,binlog Logs are in binary format , You cannot use the command of the view text tool （ such as ,cat,vi etc. ） see , While using mysqlbinlog Parse view .

summary ：binlog Add, delete and modify the record database , Do not log binary queries .

Two 、binlog What's the role ？

• It is mainly used for master-slave replication of database and incremental recovery of data .
• When data is written to the database , It will also update SQL The statement is written to the corresponding binlog In the document , This document is the one mentioned above binlog file . Use mysqldump When the backup , It's just a complete preparation of data for a period of time , However, if the database server fails suddenly after backup , This is where it comes in binlog The Journal of .

3、 ... and 、binlog Classification of log

• Binary log index file （ The file name suffix is .index） Used to record all binary files
• Binary log file （ The file name suffix is .00000*） Record everything in the database DDL and DML( In addition to data query statements select) Statement event .

Four 、binlog And redolog The difference between ？

The two are used in different ways

• binlog All changes to the table will be recorded , Including updating and deleting data , Change the table structure and so on , It is mainly used for manual data recovery , and redo log It is invisible to us , It is InnoDB Used to guarantee crash-safe The ability of , That is, after the transaction is committed MySQL If it collapses , It can ensure the persistence of transactions , That is, after the transaction is committed, its changes are permanent . One sentence summary ：binlog It is used for manual recovery of data ,redolog yes MySQL Their own use , Used to ensure transaction persistence in case of database crash .
• redo log yes InnoDB Engine specific ,binlog yes MySQL Of Server Layer , All engines can use .
• redo log Files are fixed size , It's written in cycles , When it is full, I will continue to write from the beginning , and binlog It's additional , When it's full, create a new file and then write .

5、 ... and 、 How to configure binlog？

Be careful ： stay kali In the system ,binlog The profile is located in ：

/etc/mysql/my.cnf

1、 Open profile ：

vim /etc/mysql/my.cnf

2、 Add the following code at the end of the configuration file ：

[mysqld]
log-bin=mysql-bin

3、 restart mysqld Service makes configuration effective

/etc/init.d/mysqld stop
/etc/init.d/mysqld restart

** Be careful ：** Every time the server （ database ） restart , The server will call flush logs;, Create a new binlog journal ！

4、 see binlog Whether the log is on

Get into MySQL see ：

myslq -uroot -p

mysql> show variables like 'log_%';

As shown in the above figure, it means binlog Open successfully

6、 ... and 、 frequently-used binlog Log operation command

1、 View all binlog Log list

mysql> show master logs;

2、 see master state , In the end ( newest ) One binlog The number and name of the log , And the last operational event pos The end point (Position) value

mysql> show master status;

3、flush Refresh log journal , Since then, a new number of binlog Log files

Be careful ： whenever mysqld When the service is restarted , This command will be executed automatically , Refresh binlog journal ; stay mysqldump Add... When backing up data -F Options will also refresh binlog journal ;

4、 Reset ( Empty ) all binlog journal

7、 ... and 、 see binlog Log contents

Ⅰ Use mysqlbinlog Check the command law with you

stay kali In the system ,myslq Of binlog be located /var/lib/mysql/ Under the path

Use mysqlbinlog Command view binlog Log contents , Here's one of the fragments for analysis ：

mysqlbinlog	mysql-bin.000001

..............
# at 624
#160925 21:29:53 server id 1 end_log_pos 796 Query thread_id=3 exec_time=0 error_code=0
SET TIMESTAMP=1474810193/*!*/;
insert into member(`name`,`sex`,`age`,`classid`) values('wangshibo','m',27,'cls1'),('guohuihui','w',27,'cls2')     **# Executive sql sentence **
/*!*/;
# at 796
#160925 21:29:53 server id 1 end_log_pos 823 Xid = 17          **# Execution time **
.............

Explanation of the name ：
server id 1 ： The service number of the database host ;
end_log_pos 796： sql At the end of pos node
thread_id=3： Thread number

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-R23nNCdV-1657975954231)(https://s2.loli.net/2022/07/16/UNslHQ5IiETdO9c.png)]

Example ：

 example ：mysqlbinlog -d dadong dadong-bin.000001 dadong-bin.000002 -r bin.log   ## utilize mysqlbinlog analysis binlog File to bin.log in .
 utilize mysqlbinlog  -d Parameter details the specified library binlog journal 
-d  Specify resolution dadong database ,
-r  Unloading assembly sql sentence , Specify the generated file 

Intercept by location binlog Content

Reference article ：binlog Introduce - Qinxi - Blog Garden (cnblogs.com)

Intercept by location binlog The advantage of content is accuracy , But it takes time to choose the location , for example ： To intercept dadong-bin.000009 File from location 365 To the position 465 Log , The order is as follows ：

[[email protected] ~]# mysqlbinlog dadong-bin.000009 --start-position=365 --stop-position=465 -r pos.sql
 Tips ： Start position must exist binlog in , The end position point may not exist .
 If the start position is specified , Do not specify the end position , Will intercept from the beginning to the end binlog journal ：
mysqlbinlog dadong-bin.000009 --start-position=365  -r pos.sql
 If the end position is specified , Do not specify the starting position , Then intercept everything from the beginning to the end binlog journal ：
mysqlbinlog dadong-bin.000009  --stop-position=465 -r pos.sql
 The so-called location point , Namely mysqlbinlog Parse the beginning of different lines in the file “#at  Numbers ” Identification data .

Example ：

mysqlbinlog dadong-bin.000009 --start-position=365 --stop-position=456 -r pos.sql

mysqlbinlog dadong-bin.000005 --start-position=2265 --stop-position=2552 -r pos.sql
mysqlbinlog dadong-bin.000009 --start-position=365 --stop-position=456 -r pos.sql
mysqlbinlog dadong-bin.000009 --start-position=365 -r pos.sql
mysqlbinlog dadong-bin.000009 --stop-position=456 -r pos.sql

 Interception part binlog according to pos
mysqlbinlog dadong-bin.000009 --start-position=365 --stop-position=456 -r pos.sql
mysqlbinlog dadong-bin.000009 --start-position=365 -r pos.sql
mysqlbinlog dadong-bin.000009 --stop-position=456 -r pos.sql

 Interception part binlog According to time 
mysqlbinlog dadong-bin.000009 --start-datetime='2014-10-16 17:14:15' --stop-datetime='2014-10-16 17:15:15' -r time.sql
mysqlbinlog dadong-bin.000009 --start-datetime='2014-10-16 17:14:15'  -r time.sql
mysqlbinlog dadong-bin.000009  --stop-datetime='2014-10-16 17:15:15' -r time.sql

Ⅱ stay MySQL Use in show Command query

Command format ：

mysql> show binlog events [IN 'log_name'] [FROM pos] [LIMIT [offset,] row_count];

Parameter interpretation ：

IN 'log_name' ： Specify the binlog file name ( No designation is the first binlog file )
FROM pos	 ： Specify from which pos Start from ( Do not specify is from the entire file first pos Start at )
LIMIT [offset] ： Offset ( No designation is 0)
row_count ： Total number of queries ( Not specifying is all lines )

MariaDB [test]> show binlog events in 'mysql-bin.000001'\G;

MariaDB [test]> show binlog events in 'mysql-bin.000001'\G;
*************************** 1. row ***************************
   Log_name: mysql-bin.000001
        Pos: 4
 Event_type: Format_desc
  Server_id: 1
End_log_pos: 256
       Info: Server ver: 10.5.12-MariaDB-1-log, Binlog ver: 4
*************************** 2. row ***************************
   Log_name: mysql-bin.000001
        Pos: 256
 Event_type: Gtid_list
  Server_id: 1
End_log_pos: 285
       Info: []
*************************** 3. row ***************************
   Log_name: mysql-bin.000001
        Pos: 285
 Event_type: Binlog_checkpoint
  Server_id: 1
End_log_pos: 328
       Info: mysql-bin.000001
*************************** 4. row ***************************
   Log_name: mysql-bin.000001
        Pos: 328
 Event_type: Gtid
  Server_id: 1
End_log_pos: 370
       Info: BEGIN GTID 0-1-1
*************************** 5. row ***************************
   Log_name: mysql-bin.000001
        Pos: 370
 Event_type: Intvar
  Server_id: 1
End_log_pos: 402
       Info: INSERT_ID=8
*************************** 6. row ***************************
   Log_name: mysql-bin.000001
        Pos: 402
 Event_type: Query
  Server_id: 1
End_log_pos: 565
       Info: use `test`; insert into member(`name`,`age`,`classid`) values('wangshibo',27,'cls1'),('guohuihui',27,'cls2')
*************************** 7. row ***************************
   Log_name: mysql-bin.000001
        Pos: 565
 Event_type: Xid
  Server_id: 1
End_log_pos: 596
       Info: COMMIT /* xid=440 */
7 rows in set (0.000 sec)

The above statement can change the specified binlog Log files , Return by dividing into valid event lines , And can use limit Appoint pos The starting offset of the point , Number of queries ！
Here's an example ：
a） Search for the first ( Earliest ) Of binlog journal ：

mysql> show binlog events\G;

b） Specify query mysql-bin.000002 This file ：

mysql> show binlog events in 'mysql-bin.000002'\G;

c） Specify query mysql-bin.000002 This file , from pos spot :624 Start looking up ：

mysql> show binlog events in 'mysql-bin.000002' from 624\G;

d） Specify query mysql-bin.000002 This file , from pos spot :624 Start looking up , Inquire about 10 strip （ namely 10 statement ）

mysql> show binlog events in 'mysql-bin.000002' from 624 limit 10\G;

e） Specify query mysql-bin.000002 This file , from pos spot :624 Start looking up , The offset 2 That's ok （ That is to skip in the middle 2 individual ）, Inquire about 10 strip

mysql> show binlog events in 'mysql-bin.000002' from 624 limit 2,10\G;

8、 ... and 、 utilize binlog Log recovery mysql data

Ⅰ Create test cases

1、 Get into MySQL

mysql -uroot -p 

2、 Create a test library

create database test;

3、 Create a member surface

use test;
CREATE TABLE IF NOT EXISTS `member` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(16) NOT NULL,
`age` tinyint(3) unsigned NOT NULL,
`classid` varchar(6) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

4、 See if the table was created successfully ：

show tables;

View the structure of the table ：

desc member;

5、 insert data

insert into member(`name`,`age`,`classid`) values('wangshibo',27,'cls1'),('guohuihui',27,'cls2');

View the data ：

select * from member;

Ⅱ Conduct scene simulation

1、 First the test Database backup

take test Database backup to /opt/backup/ops_$(date +%F).sql.gz In file ：

┌──(rootkali)-[/var/lib/mysql]
└─# mysqldump -uroot -p -B -F -R -x --master-data=2 test|gzip >/opt/backup/test_$(date +%F).sql.gz 
Enter password: 

Check if the backup is successful ：

ls /opt/backup/

Parameter description ：

-B： Specify database 
-F： Refresh the log 
-R： Backup stored procedures, etc 
-x： Lock table 
--master-data： Add... To the backup statement CHANGE MASTER Statements and binlog File and location information 

Because the above is used in full backup -F Options , Then, when the data backup operation starts, the system will automatically refresh log, This will automatically generate a new binlog journal , This new binlog The log will be used to record the database after backup “ Additions and deletions ” operation

2、 Insert new data in the work

insert into member(`name`,`age`,`classid`) values('demo0',22,'cls3'),('demo1',11,'cls4');

View the data ：

select * from member;

3、 If you accidentally delete test library

drop database test;

Ⅲ Restore backup data

1、 This time , Don't panic ！！！
Check out the last one first binlog journal , And record the key pos spot , Which is it pos The operation of the point leads to the destruction of the database ( Usually in the last few steps );

Back up the last one first binlog Log files ：

┌──(rootkali)-[/]
└─# cd /var/lib/mysql/ ┌──(rootkali)-[/var/lib/mysql]
└─# ls 
aria_log.00000001  ib_buffer_pool  ibtmp1             mysql-bin.000001  mysql-bin.000004  mysql_upgrade_info
aria_log_control   ibdata1         multi-master.info  mysql-bin.000002  mysql-bin.000005  performance_schema
debian-10.5.flag   ib_logfile0     mysql              mysql-bin.000003  mysql-bin.index
                                                                                                
┌──(rootkali)-[/var/lib/mysql]
└─# cp -v mysql-bin.000005 /opt/backup/
'mysql-bin.000005' -> '/opt/backup/mysql-bin.000005'
                                                                                                      
┌──(rootkali)-[/var/lib/mysql]
└─# ls /opt/backup/
mysql-bin.000005  test_2022-07-16.sql.gz

2、 Then perform a refresh log index operation , Start a new binlog Log files .

Logically speaking mysql-bin.000005 This file will not be written later , But in order to facilitate our analysis and search ops node , So let all subsequent database operations be written to the next log file .

Refresh the log index ：

mysql> flush logs;

3、 Read binlog journal , To analyze problems .
Method 1 ： Use mysqlbinlog Read binlog journal ：

cd /var/lib/mysql/
mysqlbinlog mysql-bin.000005

Method 2 ： logon server , And look at ( This method is recommended )

mysql> show binlog events in 'mysql-bin.000005';
 perhaps 
mysql> show binlog events in 'mysql-bin.000005'\G;

MariaDB [(none)]> show binlog events in 'mysql-bin.000005'\G;
*************************** 7. row ***************************
   Log_name: mysql-bin.000005
        Pos: 459
 Event_type: Query
  Server_id: 1
End_log_pos: 614
       Info: use `test`; insert into member(`name`,`age`,`classid`) values('demo0',00,'cls3'),('demo1',11,'cls4')
*************************** 8. row ***************************
.......
.......

*************************** 10. row ***************************
   Log_name: mysql-bin.000005
        Pos: 687
 Event_type: Query
  Server_id: 1
End_log_pos: 772
       Info: drop database test
*************************** 11. row ***************************
   Log_name: mysql-bin.000005
        Pos: 772
 Event_type: Rotate
  Server_id: 1
End_log_pos: 819
       Info: mysql-bin.000006;pos=4
11 rows in set (0.000 sec)

Through analysis , Causing database corruption pos The point interval is between 687-772 Between （ This is based on log intervals pos It's a node ）, Just go back to 687 Just before .

4、 Use the data backed up before , Restore to the backup at that time ：

stay kali in ：

┌──(rootkali)-[/var/lib/mysql]
└─# cd /opt/backup/ 
┌──(rootkali)-[/opt/backup]
└─# ls 
mysql-bin.000005  test_2022-07-16.sql.gz                  
             
┌──(rootkali)-[/opt/backup]
└─# gzip -d test_2022-07-16.sql.gz 
┌──(rootkali)-[/opt/backup]
└─# ls
mysql-bin.000005  test_2022-07-16.sql                                                                                                                          
┌──(rootkali)-[/opt/backup]
└─# mysql -uroot -p -v < test_2022-07-16.sql 
Enter password: 

View recovered data ：

.

But this is only to restore the data at the time of backup , The inserted data after backup has not been recovered

5、 from binlog Log recovery data
Restore the syntax of the command ：

mysqlbinlog mysql-bin.0000xx | mysql -u user name  -p password   Database name 

Explanation of common parameter options ：

--start-position=687  start pos spot 
--stop-position=772  end pos spot 
--start-datetime="2016-9-25 22:01:08"  Starting time 
--stop-datetime="2019-9-25 22:09:46"  End point 
--database=zyyshop  Specify restore only zyyshop database ( A host often has multiple databases , Only local log journal )

Not commonly used options ：

-u --user=name  User name to connect to the remote host 
-p --password[=name]  Password to connect to the remote host 
-h --host=name  Get... From the remote host binlog journal 
--read-from-remote-server  From somewhere MySQL Read on the server binlog journal 

Summary ： It's actually going to read binlog Log contents , To pass through a pipe symbol to mysql command . These orders 、 Try to write the file as absolute path ;

6、 Let the data recover completely

a） Complete recovery ( Manual required vim edit mysql-bin.000006, Take that one drop The sentence is eliminated )

cd /opt/backup/  
cp /var/lib/mysql/mysql-bin.000006 /opt/backup
mysqlbinlog /opt/backup/mysql-bin.000006 > /opt/backup/000006.sql
vim /opt/backup/000006.sql # Delete the inside drop sentence 
mysql -uroot -p -v < /opt/backup/000006.sql

reminder ：
Before full data can be recovered, the binlog File removal , Otherwise, in the process of recovery , Will continue to write statements to binlog, Finally, the incremental recovery data becomes chaotic ！
May refer to ：https://www.cnblogs.com/kevingrace/p/5904800.html

b） Appoint pos End point recovery ( Partial recovery )：
–stop-position=687 pos The end node （ According to the transaction interval , yes 687）

/usr/bin/mysqlbinlog --stop-position=687 --database=test /var/lib/mysql/mysql-bin.000005 | /usr/bin/mysql -uroot -p  -v test

c） Recover separately according to the transaction interval , recovery [ 459,614]

/usr/bin/mysqlbinlog --start-position=459 --stop-position=614 --database=ops /var/lib/mysql/mysql-bin.000005 | /usr/bin/mysql -uroot -p -v ops