[percona toolkit] series of Pt table checksum and Pt table sync data verification and repair artifacts

brief introduction

This time it mainly includes two tools , Can be in percona-tookit Found inside

• pt-table-checksum
• pt-table-sync

pt-table-checksum

# brief introduction 
pt-table-checksum It is mainly used to verify the consistency of master-slave data , One master, many followers , One master and one slave can 
 
# Common parameters 
--no-check-replication-filters       Do not check replication filtering 
--no-check-binlog-format             Do not check binlog Format , because binlog You need to use statement Format 
--replicate                          The results of the inspection are written to , Which one? db Of checksums In the table 
CREATE TABLE checksums (
   db             CHAR(64)     NOT NULL,
   tbl            CHAR(64)     NOT NULL,
   chunk          INT          NOT NULL,
   chunk_time     FLOAT            NULL,
   chunk_index    VARCHAR(200)     NULL,
   lower_boundary TEXT             NULL,
   upper_boundary TEXT             NULL,
   this_crc       CHAR(40)     NOT NULL,
   this_cnt       INT          NOT NULL,
   master_crc     CHAR(40)         NULL,
   master_cnt     INT              NULL,
   ts             TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
   PRIMARY KEY (db, tbl, chunk),
   INDEX ts_db_tbl (ts, db, tbl)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
--chunk-size                         Default 1000 One record chunk To do query
--ignore-columns                     Specify the fields that need to ignore validation , If there are more than one, use ','( comma ) separate .
--ignore-databases                   Specify the database to ignore validation , If there are more than one, use ','( comma ) separate .
--ignore-databases-regex             Specify a database that uses regular expression matching to ignore validation .
--ignore-engines                     The default value is ：FEDERATED,MRG_MyISAM Specifies the table of the storage engine type that needs to ignore validation , If there are more than one, use ','( comma ) separate .
--ignore-tables                      Ignore which table to check 
--ignore-tables-regex                Ignore which table to check , Policy expression 
--max-lag                            When the master-slave delay reaches the maximum, it stops , Default 1s
--max-load                           The load can be specified , Default threads-running=25
--progress                           speed of progress 
--tables                            check surface 
--tables-regex=user_watch_tab*       only check Which watch , Regular expressions 
--ask-pass                           Need to display input ciphertext 
--replicate-check-only               Only verify databases with inconsistent results 
--recursion-method                   The default value is ：processlist,hosts
 Specify how to get from the library .pt-table-checksum It will be executed multiple times during the verification operation REPLICA CHECKS operation .
METHOD       USES
===========  =============================================
processlist  SHOW PROCESSLIST  
hosts        SHOW SLAVE HOSTS  
cluster      SHOW STATUS LIKE 'wsrep\_incoming\_addresses'
dsn=DSN      DSNs from a table
none         Do not find slaves
==========================================================
processlist： adopt SHOW PROCESSLIST Way to find slave, It's the default way , When SHOW SLAVE HOSTS Is not available . Once the instance is running in non 3306 When on the port ,hosts The mode will change to the default mode ;
hosts： adopt SHOW SLAVE HOSTS Way to find slave,hosts Mode requires configuration from the library '--report_host' and '--report_port' These two parameters ;
cluster： Based on the cluster version Galera 23.7.3 And update ;
dsn： By reading the table from the library DSN Information to connect .
dsn It's a watch , For multiple slave When ,dsn Table best 
 
CREATE TABLE `dsns` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `parent_id` int(11) DEFAULT NULL,
  `dsn` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
);
 
# Check for inconsistencies sql
SELECT db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks
FROM percona.checksums
WHERE (
 master_cnt <> this_cnt
 OR master_crc <> this_crc
 OR ISNULL(master_crc) <> ISNULL(this_crc))
GROUP BY db, tbl;

pt-table-sync

# brief introduction 
 Used to repair inconsistent data between master and slave 
 
# Common parameters 
--replicate=  ： Designate to pass pt-table-checksum The resulting table , this 2 Every tool will be used almost all the time .
--databases=  :  Specify the database to perform synchronization , Multiple are separated by commas .
--tables=     ： Specify the table to perform synchronization , Multiple are separated by commas .
--sync-to-master ： To specify a DSN, Namely from IP, He will pass show processlist or show slave status  Go to the owner automatically .
h=127.0.0.1   ： Server address , There is... In the order 2 individual ip, For the first time Master The address of , The first 2 Next is Slave The address of .
u=root        ： Account number .
p=123456      ： password .
--print       ： Print , But don't execute orders .
--execute     ： Carry out orders .
--transaction  Specifies that tool operations use transactions instead of LOCK TABLES Statement to lock the table .
--verbose.   Print out more detailed information 
--recursion-method  Get information from the library , The default value is ：processlist,hosts  There's another one none
 
 
# Common usage 
1.Resolve differences that pt-table-checksum found on all slaves of master1:
pt-table-sync --execute --replicate test.checksum master1
2.Same as above but only resolve differences on slave1
pt-table-sync --execute --replicate test.checksum \
  --sync-to-master slave1
 
# Be careful 
 If you use non 3306 port , Will use show slave hosts To get ip, lookup slave.
 If you remove the Internet IP These can lead to missing slave, You cannot operate more than one at a time 
 You need to operate one by one in the second way above .

Actual operation

# Practical use 
 
#1. First in master Create users on , Can be connected to all slave, And authorize 
create user 'pt-table-check'@'master_ip' identified by '$pass';
grant SELECT, PROCESS, SUPER, REPLICATION SLAVE,lock tables on *.* to 'pt-table-check'@'master_ip';
grant CREATE, INSERT, UPDATE, DELETE on test.* to 'pt-table-check'@'master_ip';
 
#2.master establish dsn slave table surface , And insert data 
CREATE TABLE `test`.`dsns` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`parent_id` int(11) DEFAULT NULL,
`dsn` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
);
insert into test.dsns(dsn) values ("h=s1ip,P=6606,u=pt-table-check,p=");
insert into test.dsns(dsn) values ("h=s2ip,P=6606,u=pt-table-check,p=");
insert into test.dsns(dsn) values ("h=s3ip,P=6606,u=pt-table-check,p=");
insert into test.dsns(dsn) values ("h=s4ip,P=6606,u=pt-table-check,p=");
insert into test.dsns(dsn) values ("h=s5ip,P=6606,u=pt-table-check,p=");
insert into test.dsns(dsn) values ("h=s6ip,P=6606,u=pt-table-check,p="); 
 
#3. To begin check, Here we do regular matching 
pt-table-checksum h=master_ip,P=6606,u='pt-table-check' --no-check-replication-filters --no-check-binlog-format --ask-pass --max-lag=1 --max-load='Threads_running=25' --replicate=test.checksums --databases=jiawei_test --tables-regex=user_watch_tab* --recursion-method dsn=h=dsn Tabular ip,u='pt-table-check',p='$pass',P=6606,D=test,t=dsns
 
Checksumming jiawei_test.user_watch_tab_00000041:  22% 01:42 remain
Checksumming jiawei_test.user_watch_tab_00000041:  42% 01:22 remain
Checksumming jiawei_test.user_watch_tab_00000041:  62% 00:54 remain
Checksumming jiawei_test.user_watch_tab_00000041:  85% 00:20 remain
06-16T19:52:32      0      1 50933486          0     207       0 173.857 jiawei_test.user_watch_tab_00000041
 
#4.slave Query result table , View inconsistent 
SELECT db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks
FROM percona.checksums
WHERE (
 master_cnt <> this_cnt
 OR master_crc <> this_crc
 OR ISNULL(master_crc) <> ISNULL(this_crc))
GROUP BY db, tbl;
 
>SELECT db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks
    -> FROM test.checksums
    -> WHERE (
    ->  master_cnt <> this_cnt
    ->  OR master_crc <> this_crc
    ->  OR ISNULL(master_crc) <> ISNULL(this_crc))
    -> GROUP BY db, tbl;
+------------------+-------------------------+------------+--------+
| db               | tbl                     | total_rows | chunks |
+------------------+-------------------------+------------+--------+
| jiawei_test	   | user_watch_tab_00000000 |     258163 |      1 |
| jiawei_test      | user_watch_tab_00000003 |     244324 |      1 |
| jiawei_test 	   | user_watch_tab_00000004 |     252323 |      1 |
| jiawei_test	   | user_watch_tab_00000006 |     246460 |      1 |
| jiawei_test	   | user_watch_tab_00000011 |     248913 |      1 |
| jiawei_test	   | user_watch_tab_00000016 |     248169 |      1 |
| jiawei_test 	   | user_watch_tab_00000020 |     247681 |      1 |
| jiawei_test	   | user_watch_tab_00000021 |     241514 |      1 |
| jiawei_test 	   | user_watch_tab_00000022 |     247455 |      1 |
| jiawei_test 	   | user_watch_tab_00000025 |     249253 |      1 |
| jiawei_test 	   | user_watch_tab_00000028 |     255769 |      1 |
| jiawei_test 	   | user_watch_tab_00000034 |     247704 |      1 |
| jiawei_test	   | user_watch_tab_00000041 |     249859 |      1 |
| jiawei_test 	   | user_watch_tab_00000053 |     251265 |      1 |
| jiawei_test 	   | user_watch_tab_00000056 |     245363 |      1 |
| jiawei_test 	   | user_watch_tab_00000065 |     245756 |      1 |
| jiawei_test 	   | user_watch_tab_00000072 |     246405 |      1 |
| jiawei_test 	   | user_watch_tab_00000077 |     249218 |      1 |
| jiawei_test	   | user_watch_tab_00000084 |     253561 |      1 |
| jiawei_test 	   | user_watch_tab_00000091 |     249323 |      1 |
| jiawei_test	   | user_watch_tab_00000098 |     248128 |      1 |
+------------------+-------------------------+------------+--------+
21 rows in set (0.02 sec)
 
#5. Yes, it is pt-table-sync You can print out the data first , Simple check Next 
# If it is 3306 port , And through show slave hosts Address found 
pt-table-sync --print --replicate test.checksum h=masterip,P=6606,u='pt-table-check' --transaction --verbose --ask-pass
# If the above does not meet 
pt-table-sync --print --replicate test.checksums --sync-to-master h=s1ip,P=6606,u='pt-table-check' --ask-pass --transaction --verbose
 It can be done manually , Be careful sql-log-bin=0, Do not write binlog
 It can also be direct --execute Automatic execution 
 
#master perform 
# pt-table-sync --print --replicate test.checksums --sync-to-master h=s1ip,P=6606,u='pt-table-check' --ask-pass --transaction --verbose 2>&1 >> /data/scripts/repair.sql
Enter password for s1ip:
# Syncing via replication P=6606,h=s1ip,p=...,u=pt-table-check
# DELETE REPLACE INSERT UPDATE ALGORITHM START END EXIT DATABASE.TABLE
REPLACE INTO `jiawei_test`.`user_watch_tab_00000000`(column_names) VALUES (values) /*percona-toolkit src_db:jiawei_test src_tbl:user_watch_tab_00000000 src_dsn:P=6606,h=master_ip,p=...,u=pt-table-check dst_db:jiawei_test dst_tbl:user_watch_tab_00000000 dst_dsn:P=6606,h=s1ip,p=...,u=pt-table-check lock:1 transaction:1 changing_src:test.checksums replicate:test.checksums bidirectional:0 pid:9758 user:root hostname*/;
 
# Get the execution result and execute 
mysql -u -p -e "set sql_log_bin=0;source /data/scripts/repair.sql;"

#6.check Check whether the data is the same 
 The master and slave respectively query whether the repaired data are consistent .
 So far, the data repair is completed .
 
# remarks 
 The corresponding block will be locked during analysis .

There will be pressure during operation ？

You can see that there is basically no pressure on the database when I operate online . The tool does not have any performance impact on the database .