MySQL Log management

One 、 Classification of log

Type of log	effect
Error log	Record MySQL Server startup 、 Information such as shutdown and operation error
Transaction log	1、redo log Redo log 2、undo log Rollback log
Query log	Record all sql
Slow query log	Record operations that take longer than the specified time , If it is a full table query , Even if there is no timeout, it will be recorded
Binary log	also called binlog journal , Record the division in the database as a binary file SELECT Other operations . That is, only write operations are recorded and no read operations are recorded
relay logs	The standby database copies the binary logs of the primary database to its own relay logs , So it can be replayed locally
General log	Which account to audit 、 At what time 、 What happened

Two 、 Error log

MySQL The error log is a record of MySQL More serious warning and error messages during operation , as well as MySQL Details of each startup and shutdown . The error log uses log_error as well as log_warnings And other parameters .

Check the error log

log_warnings：
0： Indicates that no warning information is recorded
1： Indicates that warning information is logged to the error log

Greater than 1 Express " Failed connection " And when creating a new connection " Access denied " Class will also be recorded in the error log .

mysql> show variables like '%log_warnings%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_warnings  | 2     |
+---------------+-------+
1 row in set (0.00 sec)

Set error log

There are also two ways to set the error log , They are temporary settings and permanent settings .

Temporary settings

[[email protected] mysql-5.7.34]# /usr/local/mysql-5.7.34/support-files/mysql.server start --log_error=/tmp/DB-Server.localdomain.err

[[email protected] mysql-5.7.34]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.34 Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like '%log_error%';
+---------------------+--------------------------------+
| Variable_name       | Value                          |
+---------------------+--------------------------------+
| binlog_error_action | ABORT_SERVER                   |
| log_error           | /tmp/DB-Server.localdomain.err |
| log_error_verbosity | 3                              |
+---------------------+--------------------------------+
3 rows in set (0.01 sec)

mysql> 

Permanent settings

[[email protected] ~]# vim /etc/my.cnf
log-error=/var/log/mysql-error.log

[[email protected] ~]# touch /var/log/mysql-error.log
[[email protected] ~]# chown mysql.mysql /var/log/mysql-error.log

[[email protected] ~]# systemctl start mysqld

[[email protected] ~]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.34 Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like '%log_error%';
+---------------------+--------------------------+
| Variable_name       | Value                    |
+---------------------+--------------------------+
| binlog_error_action | ABORT_SERVER             |
| log_error           | /var/log/mysql-error.log |
| log_error_verbosity | 3                        |
+---------------------+--------------------------+
3 rows in set (0.00 sec)

3、 ... and 、 Log of transactions

innodb The transaction log includes redo log and undo log.redo log It's redoing the log , Provide roll forward operation ,undo log It's a rollback log , Provides rollback operations .

undo log No redo log The reverse process of , In fact, they are all logs for recovery ：

1. redo log Usually physical logs , What is recorded is the physical modification of the data page , Not how to modify a line or lines , It is used to recover the physical data page after submission ( Recover data page , And can only be restored to the location of the last submission ).
2. undo Used to roll back row records to a version .undo log It's usually a logical log , Record according to each line .

redo log

redo log It consists of two parts ：

1. Log buffer in memory (redo log buffer), This part of the log is volatile
2. Redo log file on disk (redo log file), This part of the log is persistent

Conceptually ,innodb adopt force log at commit Mechanism to achieve transaction persistence , When the transaction is committed , You must first write all transaction logs of the transaction to redo log file and undo log file Persistence in .

To ensure that each log can be written to the transaction log file , Every time log buffer In the process of writing log files, the operating system will be called once fsync operation ( namely fsync() system call ). because MariaDB/MySQL It works in user space ,MariaDB/MySQL Of log buffer In user space memory . To write to disk log file in , In the middle of the kernel space through the operating system os buffer, call fsync() The role of will be OS buffer In the log to the disk log file in .

undo log

undo log It does two things ： Provides rollback and multiple row versioning (MVCC).

When the data is modified , Not only did it record redo, The corresponding undo, If the transaction fails or rolls back for some reason , With the help of this undo Roll back .

undo log and redo log Recording physical logs is not the same , It's a logical log . Can be considered as delete When a record ,undo log A corresponding insert Record , vice versa , When update When a record , It records an opposite update Record .

When executed rollback when , You can start from undo log The logical record in reads the corresponding content and rolls back . Sometimes it is applied to row version control , through undo log To achieve ： When a read row is locked by another transaction , It can come from undo log Analyze the previous data of this line record , To provide the version information of this line , Let the user realize non lock consistent reading .

undo log It's using paragraphs (segment) The way to record , Every undo The operation occupies a while recording undo log segment. in addition ,undo log There will be redo log, because undo log We also need to achieve durable protection .

undo log Related variables

mysql> show variables like "%undo%";
+--------------------------+------------+
| Variable_name            | Value      |
+--------------------------+------------+
| innodb_max_undo_log_size | 1073741824 |
| innodb_undo_directory    | ./         |
| innodb_undo_log_truncate | OFF        |
| innodb_undo_logs         | 128        |
| innodb_undo_tablespaces  | 0          |
+--------------------------+------------+
5 rows in set (0.00 sec)

Four 、 General query log

It usually doesn't open , Because even if you start a transaction , Finally, it will be recorded without submitting , The production process runs sql quite a lot , It will take up a lot of space , Never start , It depends on the operation binlog.

[root@localhost ~]# vim /etc/my.cnf
general_log=on
general_log_file=/var/log/select.log

[root@localhost ~]# touch /var/log/select.log
[root@localhost ~]# chown mysql.mysql /var/log/select.log
[root@localhost ~]# chmod 640 /var/log/select.log

[root@localhost ~]# systemctl restart mysqld
[root@localhost ~]# mysql -uroot -p123456
mysql> show variables like '%gen%';
+----------------------------------------+---------------------+
| Variable_name                          | Value               |
+----------------------------------------+---------------------+
| auto_generate_certs                    | ON                  |
| general_log                            | ON                  |
| general_log_file                       | /var/log/select.log |
| sha256_password_auto_generate_rsa_keys | ON                  |
+----------------------------------------+---------------------+
4 rows in set (0.01 sec)

5、 ... and 、 Slow query log

MySQL The slow query log of is MySQL A type of logging provided , It is used to record in MySQL Statement with response time over threshold in , Specifically, the running time exceeds long_query_time It's worth it SQL, It will be recorded in the slow query log .long_query_time The default value is 10, It means running 10S The above sentence . By default ,Mysql The database does not start the slow query log , We need to set this parameter manually , Of course , If it's not for tuning , It is generally not recommended to start this parameter , Because opening slow query log will bring some performance impact more or less . Slow query log supports writing log records to files , It also supports writing log records to database tables .

Slow query log related parameters

MySQL Slow query related parameter interpretation ：

• slow_query_log： Open slow query log or not ,1 Open for indication ,0 Means closing .
• log-slow-queries ： Old edition （5.6 The following versions ）MySQL Database slow query log storage path . This parameter can not be set , The system will default to a default file host_name-slow.log
• slow-query-log-file： new edition （5.6 And above ）MySQL Database slow query log storage path . This parameter can not be set , The system will default to a default file host_name-slow.log
• long_query_time ： Slow query threshold , When the query time is longer than the set threshold , Log
• log_queries_not_using_indexes： Queries that are not indexed are also recorded in the slow query log （ optional ）.
• log_output： How to store logs .log_output=‘FILE’ Means to save the log to a file , The default value is ’FILE’.log_output='TABLE’ Indicates that the log is stored in the database , In this way, the log information will be written to mysql.slow_log In the table .MySQL The database supports two log storage methods at the same time , The configuration can be separated by commas , Such as ：log_output=‘FILE,TABLE’. The log is recorded in the special log table of the system , It takes more system resources than recording to a file , So for the need to enable slow query logging , Also need to be able to obtain higher system performance , Then it is suggested that priority should be given to the file .

[[email protected] ~]# vim /etc/my.cnf
[mysqld]
# Specify whether to turn on slow query log 
slow_query_log = on
# Specify the slow log file location （ Default in data）
slow_query_log_file=/var/log/slow.log
# Set the threshold of slow query ( Default 10s)
long_query_time=0.05
# Whether the slow query log without index is recorded to the log 
log_queries_not_using_indexes=ON
# The query check returns... Less than the specified line of the parameter SQL Not recorded in slow query log , Less than 100 Yes sql If the statement query is slow, it will not be recorded , Generally not used 
log_output='FILE'

Test slow log

[root@localhost ~]# cat /etc/my.cnf
[mysqld]
# Specify whether to turn on slow query log 
slow_query_log = on
# Specify the slow log file location （ Default in data）
slow_query_log_file=/var/log/slow.log
# Set the threshold of slow query ( Default 10s)
long_query_time=0.05
# Whether the slow query log without index is recorded to the log 
log_queries_not_using_indexes=ON
# The query check returns... Less than the specified line of the parameter SQL Not recorded in slow query log , Less than 100 Yes sql If the statement query is slow, it will not be recorded , Generally not used 
log_output='FILE'


[root@localhost ~]# touch /var/log/slow.log 
[root@localhost ~]# chown mysql.mysql /var/log/slow.log
[root@localhost ~]# systemctl restart mysqld

mysql>  show variables like '%slow_query%';
+---------------------+-------------------+
| Variable_name       | Value             |
+---------------------+-------------------+
| slow_query_log      | OFF               |
| slow_query_log_file | /var/log/slow.log |
+---------------------+-------------------+
2 rows in set (0.01 sec)

[root@localhost ~]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.34-log Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select sleep(3);
+----------+
| sleep(3) |
+----------+
|        0 |
+----------+
1 row in set (3.01 sec)

6、 ... and 、 Binary log

MySQL Binary log （binary log） It's a binary file , It is mainly used to record modified data or data changes that may occur MySQL sentence . Binary log （binary log） Yes... Is recorded in MySQL The database performs all the changes , And record the time of the statement 、 The execution time 、 Operation data and other additional information , But it doesn't record SELECT、SHOW Wait for those who don't change the data SQL sentence . Binary log （binary log） Mainly used for database recovery and master-slave replication , And audit （audit） operation .

Enable and set binary log

Binary logging is off by default , Start and set the binary log through the configuration file . modify my.cnf, Insert the following , And then restart mysqld service .

[mysqld]
server-id = 1                                # mysql5.7 Must be added , otherwise mysql Service startup error 
binlog_format='row'                          # binlog Working mode 
log-bin = /var/lib/mysql/mybinlog            #  Path and naming , Default in data Next 
expire_logs_days = 10                        #  Expiration time , Number of days for binary files to be automatically deleted ,0 Means not to delete 
max_binlog_size = 100M                       #  Single log file size 
binlog_rows_query_log_events=on 	           #  Open to view detailed records , The default is off

--  Turn on binglog
[root@localhost ~]# vim /etc/my.cnf
[root@localhost ~]# touch /var/lib/mysql/mybinlog
[root@localhost ~]# chown mysql.mysql /var/lib/mysql/mybinlog
[root@localhost ~]# systemctl restart mysqld

Binary log status view

System variables log_bin The value of is OFF Indicates that the binary log is not turned on （binary log）.ON Indicates that binary log is turned on （binary log）.

--  adopt show variables like 'log_bin%' View binary log settings 
mysql> show variables like 'log_bin%';
+---------------------------------+-------------------------------+
| Variable_name                   | Value                         |
+---------------------------------+-------------------------------+
| log_bin                         | ON                            |
| log_bin_basename                | /var/lib/mysql/mybinlog       |
| log_bin_index                   | /var/lib/mysql/mybinlog.index |
| log_bin_trust_function_creators | OFF                           |
| log_bin_use_v1_row_events       | OFF                           |
+---------------------------------+-------------------------------+
5 rows in set (0.01 sec)

--  View all binary log files of the current server  show binary logs / show master logs
mysql>  show binary logs;
+-----------------+-----------+
| Log_name        | File_size |
+-----------------+-----------+
| mybinlog.000001 |       154 |
+-----------------+-----------+
1 row in set (0.00 sec)

--  View the current binary log status  show master status
mysql> show master status;
+-----------------+----------+--------------+------------------+-------------------+
| File            | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-----------------+----------+--------------+------------------+-------------------+
| mybinlog.000001 |      154 |              |                  |                   |
+-----------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

Binary log switching method

Every time you restart MySQL The service also generates a new binary log file , Equivalent to binary log switching . When switching binary logs , You'll see these number It's going to increase . in addition , In addition to these binary log files , And you'll see that a DB-Server-bin.index The file of , The list of all binary log files stored in this file is also called the index of binary files .

perform flush logs You can refresh and switch binaries .

mysql> show master status;
+-----------------+----------+--------------+------------------+-------------------+
| File            | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-----------------+----------+--------------+------------------+-------------------+
| mybinlog.000001 |      154 |              |                  |                   |
+-----------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

mysql> flush logs;
Query OK, 0 rows affected (0.35 sec)

mysql> show master status;
+-----------------+----------+--------------+------------------+-------------------+
| File            | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-----------------+----------+--------------+------------------+-------------------+
| mybinlog.000002 |      154 |              |                  |                   |
+-----------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

Binary file viewing

Use show binlog events You can get binary logs .

mysql> show binlog events\G
*************************** 1. row ***************************
   Log_name: mybinlog.000001
        Pos: 4
 Event_type: Format_desc
  Server_id: 1
End_log_pos: 123
       Info: Server ver: 5.7.34-log, Binlog ver: 4
*************************** 2. row ***************************
   Log_name: mybinlog.000001
        Pos: 123
 Event_type: Previous_gtids
  Server_id: 1
End_log_pos: 154
       Info: 
*************************** 3. row ***************************
   Log_name: mybinlog.000001
        Pos: 154
 Event_type: Rotate
  Server_id: 1
End_log_pos: 200
       Info: mybinlog.000002;pos=4
3 rows in set (0.00 sec)

Print binary log to a clear text file , The file records the updated data sql, But in 5.7 The above has been encrypted

#  Print log files 
[root@localhost ~]# mysqlbinlog /var/lib/mysql/mybinlog.000001 > mysql-bin.log
#  Decrypt files 
[root@localhost ~]# mysqlbinlog --base64-output=decode-rows -v /var/lib/mysql/mybinlog.000001 > mysql-bin.log

Recover the database using binary logs

If binary logging is turned on , Data loss occurred , The database can be recovered through binary log , The grammar is as follows ：mysqlbinlog [option] filename | mysql -u user -p passwd

option There are two main parameters of --start-datetime --stop-datetime and start-position --stop-position , The former specifies the point in time for recovery , The latter specifies the location of the recovery （ Location refers to the location in the binary file # at 580 580 It's the location ）. The principle is to re execute the recorded statement , If you recover twice . It creates duplicate data .

[root@localhost mysql]# mysqlbinlog --start-position="154" /var/lib/mysql/mybinlog.000004 | mysql -uroot -p123456

Be careful , To find the point in time or location where the updated statement is inserted . If the recovered statement contains only delete, Will report a mistake 1032 error .

Temporarily stop the binary log function

You can stop the binary logging function by modifying the configuration file , But you need to restart the database ,mysql A statement is provided to stop the binary function online .

set sql_log_bin = 0       #  Stop the binary log function 
set sql_log_bin = 1       #  Turn on binary log function 

Three modes of binary logging

There are three formats for binary logs ：STATEMENT,ROW,MIXED, By the parameter binlog_format control .

1. STATEMENT Pattern （SBR）
   Each one will modify the data sql The statement will record binlog in . The advantage is that you don't need to record every one of them sql Statement and data changes for each line , Less binlog Log volume , save IO, Improve performance . The disadvantage is that in some cases （ For example, nondeterministic functions ） It will lead to master-slave The data in is inconsistent ( Such as sleep() function , last_insert_id(), as well as user-defined functions(udf) There will be problems later ).

2. ROW Pattern （RBR）
   Don't record every sql The context information of the statement , Just record which data has been modified , What's the change . And there will be no stored procedures under certain circumstances 、 or function、 or trigger The call and trigger of cannot be copied correctly . The disadvantage is that there will be a lot of logs , In especial alter table When it's time to make the journal skyrocket .

3. MIXED Pattern （MBR）
   Mixed use of the above two modes , General replication uses STATEMENT Mode save binlog, about STATEMENT Mode can not be copied by operation using ROW Mode save binlog,MySQL It will be executed according to SQL Statement to choose how to save the log .

7、 ... and 、 relay logs

From the server I/O The thread reads the binary log from the primary server and logs it to the slave server local file , And then from the server SQL The thread will read relay-log The contents of the log are applied to the slave server , So that the data of the slave server and the master server are consistent .

mysql> show variables like '%relay%';
+---------------------------+--------------------------------------------------------+
| Variable_name             | Value                                                  |
+---------------------------+--------------------------------------------------------+
| max_relay_log_size        | 0                                                      |
| relay_log                 |                                                        |
| relay_log_basename        | /usr/local/mysql-5.7.34/data/localhost-relay-bin       |
| relay_log_index           | /usr/local/mysql-5.7.34/data/localhost-relay-bin.index |
| relay_log_info_file       | relay-log.info                                         |
| relay_log_info_repository | FILE                                                   |
| relay_log_purge           | ON                                                     |
| relay_log_recovery        | OFF                                                    |
| relay_log_space_limit     | 0                                                      |
| sync_relay_log            | 10000                                                  |
| sync_relay_log_info       | 10000                                                  |
+---------------------------+--------------------------------------------------------+
11 rows in set (0.00 sec)

Detailed explanation of variables

relay_log fileName：        Specify the file name of the relay log .【 The file name is empty , Indicates that relay logging is disabled 】
relay_log_index：           Index table 
relay_log_info_file：       Record information about the relay log file 
relay_log_purge：           Specify whether to automatically delete useless relay log files 
relay_log_recovery：        Whether the relay log can be configured for automatic recovery 
relay_log_space_limit：     Specify the amount of space that the relay log can occupy （0 Means unrestricted ）

8、 ... and 、 General log

Record connection database information and all operation information .

mysql> show variables where variable_name like "%general_log%" or variable_name="log_output";
+------------------+---------------------+
| Variable_name    | Value               |
+------------------+---------------------+
| general_log      | ON                  |
| general_log_file | /var/log/select.log |
| log_output       | FILE                |
+------------------+---------------------+
3 rows in set (0.00 sec)

• general_log：OFF Indicates that the general log is turned off ,ON Indicates that the general log is enabled
• general_log_file： Indicates the general log file path
• log_output：FILE Represents a record file ,TABLE Indicates a record table ,FILE,TABLE Means to record files and tables at the same time

Test the general log

mysql> SET GLOBAL general_log = 'ON';
Query OK, 0 rows affected (0.00 sec)

mysql> SET GLOBAL log_output = 'FILE,TABLE';
Query OK, 0 rows affected (0.00 sec)

mysql> select * from mysql.general_log\G
*************************** 1. row ***************************
  event_time: 2021-10-13 21:22:23.599822
   user_host: root[root] @ localhost []
   thread_id: 2
   server_id: 1
command_type: Query
    argument: select * from mysql.general_log
*************************** 2. row ***************************
  event_time: 2021-10-13 21:22:55.783993
   user_host: [root] @ localhost []
   thread_id: 4
   server_id: 1
command_type: Connect
    argument: root@localhost on  using Socket
*************************** 3. row ***************************
  event_time: 2021-10-13 21:22:55.784466
   user_host: root[root] @ localhost []
   thread_id: 4
   server_id: 1
command_type: Query
    argument: select @@version_comment limit 1
*************************** 4. row ***************************
  event_time: 2021-10-13 21:22:58.409048
   user_host: root[root] @ localhost []
   thread_id: 2
   server_id: 1
command_type: Query
    argument: select * from mysql.general_log
*************************** 5. row ***************************
  event_time: 2021-10-13 21:23:27.197193
   user_host: root[root] @ localhost []
   thread_id: 2
   server_id: 1
command_type: Query
    argument: select * from mysql.general_log
5 rows in set (0.00 sec)