Canal realizes MySQL data synchronization

Preface
canal It's the realization of mysql The data backup , Remote disaster recovery , Important middleware such as remote data synchronization , It is widely used in actual business scenarios , This paper is based on a scenario of remote data synchronization in the project where Xiaobian is working , Use through case demonstration canal Realization mysql The process of data synchronization

Synchronization principle

As shown in the figure above , by canal Sync mysql A simple schematic diagram of the principle of data , It can be summed up as follows ：

 1.canal simulation mysql slave Interaction protocol , Pretend to be mysql slave, towards mysql master send out dump agreement .
 2.mysql master received dump request , Start pushing binary log to canal (slave)
 3.canal analysis binary log object ( Originally byte flow )
If you have configured mysql Students who read and write separate structures , Yes canal The principle of data replication is well understood , Simply speaking , Just pretend to be master The slave node , Listen and analyze binlog Log metadata to the data table of the target database

Environmental preparation
One opens binlog The log mysql service ,IP The address is : IP1
Another installed and started mysql The server's machine ,IP The address is ： IP2
canal Installation package , Include ：canal.adapter-1.1.4.tar.gz , canal.deployer-1.1.4.tar.gz
canal Installation package download address ：https://github.com/alibaba/canal/releases

Implementation steps
To realize the requirements of this article , It is mainly based on the configuration of the downloaded two installation packages , In fact the canal Students who know something about it are not unfamiliar with its implementation principle , That's the two one. springboot Service for , For developers , Just follow the official wiki Synchronization implementation provided , Just make various configurations

In the last article springboot Integrate canal In one article , We preliminarily demonstrated canal Of server End configuration process , namely deployer Configuration of , This article , Need to be right deployer and adapter Configure at the same time

deployer To configure
1、 decompression canal.deployer-1.1.4.tar.gz To the specified directory

tar -zxvf canal.deployer-1.1.4.tar.gz -C Destination file directory

2、 Get into conf Catalog , modify canal.properties

If it is msyql Synchronize to mysql, This configuration file does not need to be modified , Keep the default , The points needing attention are as follows ：

 

3、 Get into example Catalog , modify instance.properties

The main changes are as follows ：

1.canal.instance.master.address=IP:3306    #mysql source address

2.mysql Source address authorization canal Get the account name and password of the data

1.canal.instance.dbUsername=root
2.canal.instance.dbPassword=123456

4、 start-up deployer service

Enter into bin Catalog , Directly use the following script to start , Check the process after startup

adapter To configure
1、 decompression canal.adapter-1.1.4.tar.gz To the specified directory

1.tar -zxvf canal.adapter-1.1.4.tar.gz Specify the directory

2、 Get into conf Catalog , The first change application.yml file

The main changes are as follows ,

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp # kafka rocketMQ
  canalServerHost: Canal Of deployer Where the service is IP Address :11111
#  zookeeperHosts: slave1:2181
#  mqServers: 127.0.0.1:9092 #or rocketmq
#  flatMessage: true
  batchSize: 500
  syncBatchSize: 1000
  retries: 0
  timeout:
  accessKey:
  secretKey:
   srcDataSources:
     defaultDS:
       url: jdbc:mysql://mysql Source database IP:3306/shop001?useUnicode=true
       username: root
       password: root
  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: g1
      outerAdapters:
      - name: logger
      - name: rdb
        key: mysql1
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql:// Target synchronization IP:3306/shop001?useUnicode=true
          jdbc.username: root
          jdbc.password: root
 

3、 Get into rdb Catalog , The first change mytest_user.yml file

  The content of the core modified file is posted below , Some things to pay attention to include ：

dataSourceKey: defaultDS
destination: example
groupId: g1
outerAdapterKey: mysql1
concurrent: true
dbMapping:
  database: shop001    #mysql Source database name
  table: user_info       #mysql The name of the table under the source database
  targetTable: shop001.tar_user        # Synchronize to target database - Data table name
  targetPk:
    id: id
  mapAll: true        # If the field structure of the source table and the target table are completely consistent , Directly open this configuration , Below targetColumns You can comment out
  #targetColumns:
  #  id:
  #  name:
  #  role_id:
  #  c_time:
  #  test1:
  #etlCondition: "where c_time>={}"
  #commitBatch: 3000 # The size of the batch submission
 

4、 start-up adapter service

Get into bin Under the table of contents , Use the following startup script to start , Then check the service process

  Before the test , We are in IP1 and IP2 Installed above mysql Under the database user_info Table and tar_user surface , Now the need is , Data should be taken from IP1:user_info => IP2: tar_user Table synchronization ,

  The current state is , Both tables under the two libraries are empty , Perform the following data insertion sql , to user_info Table inserts a piece of data

1. INSERT INTO `shop001`.`user_info` (`id`, `name`, `sex`) VALUES ('1', 'xiaoma', 'male');

You can see , The data is inserted into tar_user In the table , Of course , When you delete 、 Modification and other operations , It will also synchronize the past

  Based on the above operation , We have completed a use canal Realization mysql The process of data synchronization , The government provides many synchronization schemes , For example, data can be transferred from mysql Synchronize to mysql,es,redis wait , Interested students can continue to study ！

original text ：canal Realization mysql Data synchronization _ Uncle xiaomenong's blog -CSDN Blog _canal mysql