Data synchronization

1. Application scenarios

• Business data has developed to a certain level , Most of the hot and cold data need to be transferred from the familiar DB Migrate to other storage for complex queries and analysis
• After the sub-library sub-table , Some report class queries cannot work , It needs to be summarized to a single database table for operation
• The sub database and sub table have multiple dimensions , Need to copy multiple copies of data to achieve redundancy
• Through pseudo data sharing （ There's no way to introduce MQ、 Cannot share library table ） Business transformation
• Slow storage –>Cache Synchronization between
• Live data migration /scheme change
• Import data import data

A lot of times ,DataBus What is provided is just a tool set . To complete the final function , Most need to introduce other components , Such as MQ、JOB And so on . meanwhile , Most data synchronization tools need to have standard database support . therefore , Before you get busy synchronizing data , It is necessary to conduct a centralized data governance on the legacy data .

General data synchronization , You can apply driver double write ： The application layer writes data to the database or multiple stores at the same time . Because the code is in your own hands , This approach is intuitively simple and controllable . But the consistency problem it introduces will be a very big reduction , Because there is no complex coordination agreement （ For example, a two-phase submission agreement or paxos Algorithm ）, When something goes wrong , It is difficult to ensure that multiple stores are in the same locked state . Both systems need to perform exactly the same write operation , And complete the serialization in the same order . If the write operation is conditional or has partially updated semantics , Then things will become more troublesome .

Based on database logs ： Use the database as the only real data source , And extract the changes from the transaction or commit log . This solves the consistency problem , But it's hard to achieve ,MySQL Such databases have proprietary transaction log formats and replication redundancy solutions , It is difficult to guarantee the availability of the upgraded version . Because the solution is to deal with data changes initiated by application code &