Shardingsphere proxy 4.1 sub database and sub table

One 、ShardingSphere-Proxy Core concept of

• ShardingSphere-Proxy Concept

  Official address ：https://shardingsphere.apache.org/index_zh.html

  ShardingSphere-Proxy Is the agent of the database , Pictured ：

  

• ShardingSphere-Proxy Which databases are the main agents

  Default agent ：Mysql、PostSql

• Achieve the purpose of the agent

  1. It is mainly to complete the division of warehouse and table

  2. Read and write separation

     These two are also ShardingSphere-Proxy The two core functions of .

• Sub database and sub table

  • Concept and purpose of sub database

    • Concept

      The tables in the database are stored in different databases ; Pictured ：

      

    • Purpose

      Prevent resource competition among multiple tables in a library 【CPU、 Memory 】, Performance degradation .

  • Concept and purpose of sub table

    • Concept

      Divide a table in the database into multiple tables , Pictured ：

      

    • Purpose

      Split table is to solve the problem that the amount of data in the table is too large , Improve the performance of user query and data addition .

      such as ： With mysql Database, for example , When users add data, they will pass mysql Of InnoDB The engine stores data ,InnoDB The engine should ensure that the performance of data is within a certain range , The peak value of the largest amount of data in the table is 2000w, If exceeded 2000W Then the performance of adding data will decline , So we're going to surpass 2000W The table of data volume is split into multiple tables , Only in this way can we ensure the user experience .

    • defects

      1. Too much concurrency , There will be resource competition in the table [CPU、 Memory ] The problem of , This leads to performance degradation , The user experience becomes worse .

         Solution ： sub-treasury

• Sub database and sub table

  • Purpose

    Solve the problems of table resource competition and large amount of data .

Two 、ShardingSphere-Proxy Application scenarios of

• scene

  Both individual projects and micro service projects can use sub database and sub table .

3、 ... and 、ShardingSphere-Proxy Distribution table landing

• Tools

  ShardingSphere-Proxy

• programme

  • In process

    Pictured ：

    

    • defects
      1. Resource competition .
      2. Abnormal influence problem .

  • Out of process 【 recommend 】

    Pictured ：

    

    • defects
      1. The problem of large amount of maintenance .
      2. The performance is weaker than that in the process .
         • It can be placed in the intranet for communication 【docker】

• Realization

  • Conditions

    • Mysql database edition ：5.7

    • ShardingSphere-Proxy

      • Download address of online disk

         link ：https://pan.baidu.com/s/15yUIDQOdDDwUtVLNxNa9Cg 
         Extraction code ：3hp3
        

    • Java Of JDK

      • Download address of online disk

         link ：https://pan.baidu.com/s/1A-ksNN0YicT3hXjFscGGwA 
         Extraction code ：r9e0
        

    • download Mysql The connection driver of Put the file in the root directory lab Under the folder

      Download address of online disk ：

       link ：https://pan.baidu.com/s/1924iUe7wxGpStAzxxv2K3g 
       Extraction code ：jy7z
      

  • To configure

    1. table

       • To configure

         1. config-sharding.yaml Fragmented configuration file

            # 3、 Create client connection Library  hmms： Virtual database name 【 It's better to have the same name as the real database 】  stay server.yaml name 
            schemaName: hmms
            
            # 1、 Connect mysql
            dataSources:
              hmmsdatasources-0:  # The name of the node   Customize 
                url: jdbc:mysql://127.0.0.1:3306/ Real database name ?serverTimezone=UTC&useSSL=false
                username:  Database user name 
                password:  Database password  
                connectionTimeoutMilliseconds: 30000
                idleTimeoutMilliseconds: 60000
                maxLifetimeMilliseconds: 1800000
                maxPoolSize: 50
                
            # 2、 Fragmentation rule 
            shardingRule:
              tables: # surface 
                user: # Logical table name   Which table to divide 
                  actualDataNodes: hmmsdatasources-0.user-${
                          0..1} # Divide it into several tables   This is two tables  #hmmsdatasources-0： The name of the node  
                  tableStrategy: # The data is divided into tables 
                    inline:
                      shardingColumn: useid # Sub table field 
                      algorithmExpression: user-${
                          useid % 2} # Yes useid Die taking table  
                 # Create multiple tables 
                # Table name : # Logical table name   Which table to divide 
                  #actualDataNodes: hmmsdatasources-0. Table name -${0..1} # Divide it into several tables   This is two tables  #hmmsdatasources-0： The name of the node  
            

         2. server.yaml

            authentication:
              users:
                root: # Database user name 
                  password:  Data password 
                sharding:
                  password: sharding 
                  authorizedSchemas: hmms
            

         3. ShardingSpere-Proxy

            Run the command

            # root directory bin To execute a command under a document 
             start.bat
            

            The operation result is as shown in the figure ：

            

         4. MySql database

            • New real database name

              SET NAMES utf8mb4;
              SET FOREIGN_KEY_CHECKS = 0;
              
              -- ----------------------------
              -- Table structure for user
              -- ----------------------------
              DROP TABLE IF EXISTS `user`;
              CREATE TABLE `user`  (
                `useid` int(11) NOT NULL,
                `usenam` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT ' Login name ',
                `usepwd` varchar(100) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT ' The login password ',
                `usestate` int(11) NULL DEFAULT 2 COMMENT '-1： Delete 1： Cancellation  2： normal  3： Report the loss of ',
                `usekey` varchar(100) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT ' User secret key ',
                `usetel` varchar(11) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT ' User's mobile phone ',
                `createbyid` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT ' Add by ',
                `createbytime` datetime(0) NULL DEFAULT NULL COMMENT ' Add the time ',
                `modifybyid` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT ' Modifier ',
                `modifybytime` datetime(0) NULL DEFAULT NULL COMMENT ' Modification time ',
                PRIMARY KEY (`useid`) USING BTREE
              ) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
              
              SET FOREIGN_KEY_CHECKS = 1;
              

              Pictured ：

              

            • newly build 3307 Virtual database connection

              Pictured ：

              

            • Delete table

              Delete 3306 in hmms Library user surface , And execute the script of creating a new table in the virtual database , The operation results are as follows ：

              Virtual database ：

              

              Real database

              

            • Add data

              Add two pieces of data to the virtual database , The operation results are as follows ：

              

              Real database Table 1 , The operation results are as follows ：

              

              Real database Table two , The operation results are as follows ：

              

    2. sub-treasury

       • To configure

         # 3、 Create client connection Library 
         schemaName: hmms
         
         # 1、 Connect mysql
         dataSources:
           hmmsdatasources-0: # Real database 0
             url: jdbc:mysql://127.0.0.1:3306/hmms-0?serverTimezone=UTC&useSSL=false
             username:  user name 
             password:  password  
             connectionTimeoutMilliseconds: 30000
             idleTimeoutMilliseconds: 60000
             maxLifetimeMilliseconds: 1800000
             maxPoolSize: 50
           hmmsdatasources-1: # Real database 1
             url: jdbc:mysql://127.0.0.1:3306/hmms-1?serverTimezone=UTC&useSSL=false
             username:  user name 
             password:  password  
             connectionTimeoutMilliseconds: 30000
             idleTimeoutMilliseconds: 60000
             maxLifetimeMilliseconds: 1800000
             maxPoolSize: 50 
             
         # 2、 Fragmentation rule 
         shardingRule:
           tables: # surface 
             user: # Logical table name 
               actualDataNodes: hmmsdatasources-${
                     0..1}.user # table 
               tableStrategy: # The data is divided into tables 
                 inline:
                   shardingColumn: useid # Sub table field 
                   algorithmExpression: user-${
                     useid % 2} # Yes useid Die taking table  
           defaultDatabaseStrategy: #  Data sub database strategy 
             inline:
              shardingColumn: useid # Sub database field 
              algorithmExpression: hmmsdatasources-${
                     useid % 2}  # Yes Id Take the mold branch library productdatasources-0 
         

    3. Sub database and sub table

       • To configure

         # 3、 Create client connection Library 
         schemaName: hmms
         
         # 1、 Connect mysql
         dataSources:
           hmmsdatasources-0: # Real database 0
             url: jdbc:mysql://127.0.0.1:3306/hmms-0?serverTimezone=UTC&useSSL=false
             username:  user name 
             password:  password  
             connectionTimeoutMilliseconds: 30000
             idleTimeoutMilliseconds: 60000
             maxLifetimeMilliseconds: 1800000
             maxPoolSize: 50
           hmmsdatasources-1: # Real database 1
             url: jdbc:mysql://127.0.0.1:3306/hmms-1?serverTimezone=UTC&useSSL=false
             username:  user name 
             password:  password  
             connectionTimeoutMilliseconds: 30000
             idleTimeoutMilliseconds: 60000
             maxLifetimeMilliseconds: 1800000
             maxPoolSize: 50    
             
         # 2、 Fragmentation rule 
         shardingRule:
           tables: # surface 
             user: # Logical table name 
               actualDataNodes: hmmsdatasources-${0..1}.user-${0..1} # table 
               tableStrategy: # The data is divided into tables 
                 inline:
                   shardingColumn: useid # Sub table field 
                   algorithmExpression: user-${useid % 2} # Yes useid Die taking table 
           defaultDatabaseStrategy: #  Data sub database strategy 
             inline:
              shardingColumn: useid # Sub database field 
              algorithmExpression: hmmsdatasources-${useid % 2}  # Yes Id Take the mold branch library productdatasources-0  
         

Four 、ShardingSphere-Proxy Operation principle

• The overall architecture

  

  in total 6 Stages ：

  1、Database Adaptors： The choice of database

  2、SQL Parser： analysis sql

  3、SQL Router：sql route Which real database to execute

  4、SQL Rewriter：sql Optimized rewriting The core Guaranteed performance

  5、SQL Executor Engine： perform sql sentence Get results from real database

  6、Result Merger： The results merge Get results from multiple tables

5、 ... and 、ShardingSphere_Proxy Fragmentation principle

• The concept of fragmentation

  Is to slice data into different tables .

• Patch key

  The fragment key is the field in the table . It is divided according to what field .

• Sharding algorithm

  According to rules 【 Sharding algorithm 】 Press the fragment key to divide the data into different tables .

  • Modular algorithm

    • defects

      Only numeric types can be used

  • hash+ modulus

    • If the fragment key is of character type , Just use hash+ Take the mold and divide it .

      Math.abs( Patch key .hashCode()%2)