当前位置：网站首页>Demonstrate to Xiaobai the case of sub database and sub table

Demonstrate to Xiaobai the case of sub database and sub table

2022-06-26 16:57:00 【Tianweichang】

Official account “Java Back end technology stack ”

reply “000” Get the necessary e-books for programmers

Hello everyone , I'm Lao Tian , Invited by the little friends in the group , Make a case of sub database and sub table , In this way, many people who haven't used sub databases and sub tables also have a bottom , Otherwise, you will always see articles about various concepts and solutions on the Internet .

explain ： Because it's for Xiaobai , So don't spray , It is suggested to go out and turn left to learn more awesome skills .

demand

Because the user table is too large , Take relevant SQL Optimize , Still not satisfied , Therefore, it is now divided into databases and tables .

database ：my-sharding

Database table ：t_user

Build the predicative sentence as follows ：

DROP TABLE IF EXISTS `t_user`;
CREATE TABLE `t_user` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `user_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
  `age` int NOT NULL,
  `gender` int NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8;

There are usually two schemes for dividing databases and tables ：

Split Vertically
Horizontal split

Let's demonstrate horizontal splitting , General train of thought ：

adopt t_user Tabular id Conduct hash, And then take the module with the number of databases , Get the corresponding database .
adopt hash Value and the number of tables in each database , Get the corresponding table name .

Create databases and tables

Join a 2000 Ten thousand data , So for the convenience of demonstration , We will tentatively divide it into five Libraries , Each database corresponds to five tables .

Ideal state ：2000 ten thousand /5/4, So each database gets 400 ten thousand , Score per meter 80 ten thousand .
All in all , After the sub-library sub-table , The database and tables of each table are not of the same order of magnitude as the previous ones .

Five databases ：

Each database has five tables ：

Build the predicative sentence as follows ：


DROP TABLE IF EXISTS `t_user_0`;
CREATE TABLE `t_user_0` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `user_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
  `age` int NOT NULL,
  `gender` int NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8;

DROP TABLE IF EXISTS `t_user_1`;
CREATE TABLE `t_user_1` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `user_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
  `age` int NOT NULL,
  `gender` int NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;

DROP TABLE IF EXISTS `t_user_2`;
CREATE TABLE `t_user_2` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `user_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
  `age` int NOT NULL,
  `gender` int NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `t_user_3`;
CREATE TABLE `t_user_3` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `user_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
  `age` int NOT NULL,
  `gender` int NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=8 DEFAULT CHARSET=utf8;


DROP TABLE IF EXISTS `t_user_4`;
CREATE TABLE `t_user_4` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `user_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
  `age` int NOT NULL,
  `gender` int NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8;

Project creation

Use the technology stack ：JDK8+MySQL+Spring Boot +Mybatis +Shardingsphere +Druid

maven Related dependencies ：

<dependency>
    <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
 <groupId>org.mybatis.spring.boot</groupId>
 <artifactId>mybatis-spring-boot-starter</artifactId>
 <version>2.1.0</version>
</dependency>
<dependency>
     <groupId>org.mybatis</groupId>
  <artifactId>mybatis</artifactId>
  <version>3.5.2</version>
</dependency>

<dependency>
  <groupId>mysql</groupId>
  <artifactId>mysql-connector-java</artifactId>
  <version>8.0.16</version>
  <scope>runtime</scope>
</dependency>
<dependency>
      <groupId>com.github.pagehelper</groupId>
   <artifactId>pagehelper-spring-boot-starter</artifactId>
   <version>1.2.3</version>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-test</artifactId>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.apache.shardingsphere</groupId>
   <artifactId>sharding-jdbc-spring-boot-starter</artifactId>
   <version>4.0.1</version>
</dependency>
<dependency>
   <groupId>com.alibaba</groupId>
   <artifactId>druid</artifactId>
   <version>1.1.17</version>
</dependency>
<dependency>
   <groupId>com.google.guava</groupId>
   <artifactId>guava</artifactId>
   <version>29.0-jre</version>
</dependency>

The relevant configurations of the configuration file are as follows ：


server.port=9002
mybatis.mapper-locations=classpath:/mapper/*.xml
# mybatis.type-aliases-package=com.neutral.idmapping.dbshard.pojo

#####  Connection pool configuration  #######
#  Filter settings （ first stat Very important , If you don't, you won't be able to monitor SQL）
spring.datasource.druid.filters=stat,wall,log4j2
##### WebStatFilter To configure  #######
# Enable StatFilter
spring.datasource.druid.web-stat-filter.enabled=true
# Add filtering rules 
spring.datasource.druid.web-stat-filter.url-pattern=/*
# Rule out unnecessary url
spring.datasource.druid.web-stat-filter.exclusions=*.js,*.gif,*.jpg,*.png,*.css,*.ico,/druid/*
# Turn on session Statistical function 
spring.datasource.druid.web-stat-filter.session-stat-enable=true
# default sessionStatMaxCount yes 1000 individual 
spring.datasource.druid.web-stat-filter.session-stat-max-count=1000
#spring.datasource.druid.web-stat-filter.principal-session-name=
#spring.datasource.druid.web-stat-filter.principal-cookie-name=
#spring.datasource.druid.web-stat-filter.profile-enable=

##### StatViewServlet To configure  #######
# Enable built-in monitoring page 
spring.datasource.druid.stat-view-servlet.enabled=true
# Address of built-in monitoring page 
spring.datasource.druid.stat-view-servlet.url-pattern=/druid/*
# close  Reset All  function 
spring.datasource.druid.stat-view-servlet.reset-enable=false
# Set login user name 
spring.datasource.druid.stat-view-servlet.login-username=admin
# Set login password 
spring.datasource.druid.stat-view-servlet.login-password=admin

spring.shardingsphere.props.sql.show=false
# Database name 
spring.shardingsphere.datasource.names=dp0,dp1,dp2,dp3,dp4
#datasource
spring.shardingsphere.datasource.dp0.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.dp0.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.dp0.url=jdbc:mysql://localhost:3306/my-sharding_0?useUnicode=true&characterEncoding=utf-8&serverTimeZone=CTT&allowPublicKeyRetrieval=true&serverTimezone=UTC
spring.shardingsphere.datasource.dp0.username=root
spring.shardingsphere.datasource.dp0.password=123456

 ---------- The same code is not posted here -------
#  Corresponding  dp1、dp2、dp3、dp4  And above dp0 Configuration is similar to , The difference is that the database name is different 
#  Because I used the demo of creating multiple databases locally , There is no need to repeat here 

#actual-data-nodes
# Here is the configuration of all   library . surface   Set 
# For example, the configuration here means  dp0.data_0 , dp0.data_1 ,dp0.data_2 , ...
# This abbreviation uses shardingsphere  Officially recommended grammar 
#t_user  Logical table name    stay UserMapper.xml Use in 
spring.shardingsphere.sharding.tables.t_user.actual-data-nodes=dp$->{0..4}.t_user_$->{0..4}
#table
# Set to data Middle field id As the standard of sub table , Then you will id As a parameter, it is passed into our custom table splitting method configured below for example 
spring.shardingsphere.sharding.tables.t_user.table-strategy.standard.sharding-column=id
spring.shardingsphere.sharding.tables.t_user.table-strategy.standard.precise-algorithm-class-name=com.tian.shardingdemo.common.TableShardingAlgorithm
#database
# Set to data Middle field id As the standard of sub database , Then you will id Pass it as a parameter to our customized sub database method configured below for specific operations 
spring.shardingsphere.sharding.tables.t_user.database-strategy.standard.sharding-column=id
spring.shardingsphere.sharding.tables.t_user.database-strategy.standard.precise-algorithm-class-name=com.tian.shardingdemo.common.DbShardingAlgorithm

Two partition classes of sub database and sub table ：

/**
 *  sub-treasury 
 */
public class DbShardingAlgorithm implements PreciseShardingAlgorithm<Long> {

    private Logger logger = LoggerFactory.getLogger(DbShardingAlgorithm.class);

    @Override
    public String doSharding(Collection<String> availableTargetNames, PreciseShardingValue<Long> shardingValue) {
        String databaseName = availableTargetNames.stream().findFirst().get();
        for (String dbName : availableTargetNames) {
            //shardingValue.getValue() Is the incoming value of the configuration 
            // What we choose here is afferent sql Medium id Value of field 
            String targetDbName= "dp" + genderToTableSuffix(shardingValue.getValue());
            if (dbName.equals(targetDbName)) {
                // Match to the corresponding database , such as  dp0
                // This database name corresponds to the... Configured at the data source dp0,dp1,...
                logger.info(" Database name =" + dbName);
                databaseName = dbName;
            }
        }
        return databaseName;
    }

    private String genderToTableSuffix(Long value) {
        // take id The value of the field goes to hash Value and then de modular operation to get the number of sub library （ It's just an algorithm ）
        int i = Hashing.murmur3_128(1823977).newHasher().putString(String.valueOf(value), Charsets.UTF_8).hash().asInt();
        //hash And the number of tables 
        return String.valueOf(Math.abs(i) % 5);
    }
}
/**
 *  table 
 */
public class TableShardingAlgorithm implements PreciseShardingAlgorithm<Long> {
    private Logger logger = LoggerFactory.getLogger(TableShardingAlgorithm.class);

    @Override
    public String doSharding(Collection<String> availableTargetNames, PreciseShardingValue<Long> shardingValue) {
        String table = availableTargetNames.stream().findFirst().get();
        String targetName = "t_user_" + genderToTableSuffix(shardingValue.getValue());
        for (String tableName : availableTargetNames) {
            // Check whether the calculated table name exists 
            if (tableName.equals(targetName)) {
                logger.info(" Table name = " + tableName);
                table = tableName;
            }
        }
        return table;
    }

    private String genderToTableSuffix(Long value) {
        // Figure out one. hash value  int type 
        int i = Hashing.murmur3_128(8947189).newHasher().putString(String.valueOf(value), Charsets.UTF_8).hash().asInt();
        //hash And the number of tables 
        return String.valueOf(Math.abs(i) % 5);
    }
}

The following is the business part code , First look at UserMapper.xml Content ：

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper
        PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
        "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.tian.shardingdemo.mapper.UserMapper">
    <resultMap id="User" type="com.tian.shardingdemo.entity.User">
        <id column="id" property="id"/>
        <result column="user_name" property="userName"/>
    </resultMap>
    <insert id="insert">
        INSERT INTO t_user (id, user_name,age,gender) VALUES ( #{id},#{userName},#{age},#{gender}
        );
    </insert>
    <select id="selectUserById" resultMap="User">
        select * from t_user
        <where>
            <if test="id != null">
                id = #{id}
            </if>
        </where>
    </select>

    <update id="updateAuthorIfNecessary">
        update t_user
        <trim prefix="SET" suffixOverrides=",">
            <if test="userName != null and userName != ''">
                `user_name` = #{userName},
            </if>
            <if test="gender != null and gender != 0">
                gender = #{gender},
            </if>
            <if test="age != null and age != 0">
                age = #{age},
            </if>
        </trim>
        where id=#{id}
    </update>
</mapper>

UserMapper Interface ：

import com.tian.shardingdemo.entity.User;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import org.springframework.stereotype.Repository;

@Mapper
@Repository
public interface UserMapper {

    User selectUserById(@Param("id") Long id);

    int updateAuthorIfNecessary(User user);

    int insert(User user);
}

In order to better demonstrate , I joined here controller Layer and the service layer , This is also a common development routine .

service The layer code is as follows ：

public interface IUserService {

    User selectUserById(Long id);

    void add(Long id);
}
@Service
public class UserServiceImpl implements IUserService {

    @Resource
    private UserMapper userMapper;
    @Override
    public User selectUserById(Long id) {
        return userMapper.selectUserById(id);
    }

    @Override
    public void add(Long id) {
        User user = new User();
        user.setAge(22);
        user.setGender(1);
        user.setId(id);
        user.setUserName("tian" + id);
        userMapper.insert(user);
    }
}

controller The layer code is as follows ：

@RestController
@RequestMapping
public class UserController {

    @Resource
    private IUserService userService;

    @RequestMapping(value = "/user/{id}", method = RequestMethod.GET)
    public User selectUserById(@PathVariable("id") Long id) {
        return userService.selectUserById(id);
    }

    @PostMapping("/add")
    public Object add(@RequestBody Map<String,Long> params) {
        Long id = params.get("id");
        userService.add(id);
        return "ok";
    }
}

Finally, the startup class of the project ：

@SpringBootApplication
@MapperScan({"com.tian.shardingdemo.mapper"})
public class ShardingDemoApplication {

 public static void main(String[] args) {
  SpringApplication.run(ShardingDemoApplication.class, args);
 }
}

Start project , Successful launch ：

Let's demonstrate the new data and query .

Add data to the database

First, add data to the database , What we use here is IDEA in restful Tools ：

Background log ：

Then check the database table ：

Here we are , Our data is still in the database , Let's demonstrate data query .

Data query

Input in browser ：

http://localhost:9002/user/7

Return the data ：

{"id":7,"userName":"tian7","age":22,"gender":1}

Background log ：

From the log and the returned results, we can see , We have correctly selected the corresponding database and table for us , such , A database / table query is successful .

summary

There are not many concepts in this article , Direct use case demonstration . Related conceptual articles , There are also articles on the solution of sub database and sub table , There are piles of... On the Internet , If you are interested, you can refer to .

Reference resources ：http://01vh0.cn/mLQwN