How to correctly divide data into databases and tables

If there is enough data , You need separate databases and tables to store data , It is difficult to judge to a certain extent , Overall speaking ,

Data volume ：MySQL The database is 500w-1000w The performance is quite good when it comes to , The leaflet expresses to 2000W（ If the server configuration is good ）sql optimized , Large amount of data , When inserting or combining queries frequently , Slow down , It is necessary to divide the tables .

disk ： If a database stores more data , The disk of a server becomes a bottleneck , This is the time , We need to consider the distribution of funds

Database link ： If a database instance has too many links , It's easy to reach the service limit , At this time, it is necessary to divide the databases and tables , Of course , Or by introducing Redis The form of cache , Stand in front of me , You can lower the link of the server

There are generally two ideas for dividing the database into tables ：

1. Modify the code , Let the code link the corresponding database and query the corresponding table .

Common sub table 、 Common strategies of sub base

Distribute equally hash(object)%N( For simple architecture ), This method may encounter if a user has too much data , It will cause the problem of data skew .

  Distribute by weight and poll evenly , Good idea , But it will increase the complexity of the code .

  Allocate by business , ditto .

  In accordance with consistency hash Algorithm to allocate ( It is suitable for cluster architecture , Adding and deleting nodes in the cluster will not cause data loss , Easy data migration ).

2. Using database middleware , The function of dividing databases and tables can be realized without adjusting the code , But the general middleware will have some restrictions .

Common middleware for sub database and sub table

At present, there are several basic applications ,

• TDDL client Layer scheme
• Sharding-jdbc client Layer scheme
• Mycat proxy Layer scheme
• Cobar proxy Layer scheme

1、TDDL 

Developed by Taobao team , Belong to client Layer scheme . Support basic crud Grammar is separated from reading and writing , But does not support join、 Multi table query and other Syntax .

2、Sharding-jdbc

Be open source , Belong to client Layer scheme , It has now been renamed ShardingSphere.SQL Grammar support is also more , There are not too many restrictions , Support sub database and sub table 、 Read / write separation 、 Distributed id Generate 、 Flexible business （ Try your best to deliver 、TCC Business ）.

3、Cobar

Ali b2b Team development and open source , Belong to proxy Layer scheme , It's between application server and database server . Application through JDBC Drive access Cobar colony ,Cobar according to SQL And the rules of sub base SQL Break it down , And then distribute it to MySQL Cluster different database instances .

4、Mycat

be based on Cobar Transformed , Belong to proxy Layer scheme , Supported functions are complete , The community is active .

The introduction of these middleware will bring new problems . If it is Modify the code , Will introduce code complexity , Make the code more complex . If middleware is used , It also introduces problems , For example, performance degradation , Cost of O & M , Wait . I'm sure they won't be so happy .

Is there a better solution ？

There must be , Let's introduce Tencent cloud's TDSQL ：

TDSQL MySQL edition （TDSQL for MySQL） It is a distributed database product created by Tencent , With strong consistency and high availability 、 Global deployment architecture 、 Distributed horizontal expansion 、 High performance 、 Enterprise level security and other features , At the same time provide intelligence DBA、 Automated operations 、 Monitoring alarm and other supporting facilities , To provide customers with a complete distributed database solution . at present TDSQL Has exceeded 500+ Government, enterprise and financial institutions provide public and private cloud services for databases , Customer coverage bank 、 insurance 、 negotiable securities 、 online finance 、 charging 、 Third party payment 、 The Internet of things 、 Internet +、 Government affairs and other fields .TDSQL MySQL With its high-quality products and services , Obtained a number of international and national certification , Has been the customer and the industry's unanimous recognition .