Table partition of MySQL

mysql Partition table Introduction

Encounter such a problem in work ,mysql The table in the is large , The query is very slow , Using partition table solves this problem .
Partitioning is not done at the storage engine level , So it's not just InnoDB The storage engine supports partitioning , Common storage engines MyISAM、NDB And so on . But not all storage engines support , Such as CSV、FEDORATED、MERGE Partition is not supported . Before using this partition function , You should be aware of the partition support of the selected storage engine .
MySQL The partition type supported by database is horizontal partition （ It refers to assigning records of different rows in the same table to different physical files ）, Vertical partitioning is not supported （ It refers to assigning records of different columns in the same table to different physical files ）. Besides ,MySQL The partition of the database is the index of the office area , A partition holds both data and indexes . And global partitioning means , The data is stored in each partition , But the index of all the data is placed in one object . at present ,MySQL The database does not support global partitioning yet .

Judge the present Mysql Partition support or not

MySQL The database is 5.1 Support for partitions has been added to the version

mysql> show variables like '%partition%';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| have_partitioning | YES   |
+-------------------+-------+
1 row in set (0.00 sec)

mysql Partition type of

1. range Partition

This is the most commonly used partition , This is also the partition type I used this time . The fields of the table are as follows ：

MariaDB [zabbix]> desc history;
+--------+---------------------+------+-----+---------+-------+
| Field  | Type                | Null | Key | Default | Extra |
+--------+---------------------+------+-----+---------+-------+
| itemid | bigint(20) unsigned | NO   | MUL | NULL    |       |
| clock  | int(11)             | NO   |     | 0       |       |
| value  | double(16,4)        | NO   |     | 0.0000  |       |
| ns     | int(11)             | NO   |     | 0       |       |
+--------+---------------------+------+-----+---------+-------+

Table partition I follow clock Fields are partitioned , The data of each day is divided into a region , At present, it is divided into the back 90 Partition of days , Partition shell The script is as follows ：

#!/bin/bash

# Configure environment variables 
SERVICEROBOT_USER="root"
SERVICEROBOT_PWD="654321"
SERVICEROBOT_DB="zabbix"
SERVICEROBOT_PORT="3306"
SERVICEROBOT_HOST="localhost"
MYSQL_BIN="mysql"

# Historical data retention time , Unit day 
HISTORY_DAYS=90

# Trend data retention time , Unit month 
TREND_MONTHS=12

HISTORY_TABLE="history history_log history_str history_text history_uint"
TREND_TABLE="trends trends_uint"

#MYSQL Connection command 
MYSQL_CMD=$(echo ${MYSQL_BIN} -u${SERVICEROBOT_USER} -p${SERVICEROBOT_PWD} -P${SERVICEROBOT_PORT} -h${SERVICEROBOT_HOST} ${SERVICEROBOT_DB})
echo ${MYSQL_CMD}


EXISTED_DATA_PARTITION_SQL=()
PARTITION_SQL=""


function create_partitions_history_with_existed_data() {
	#existed data
	index=0
	for((item=${HISTORY_DAYS};item>=0;item--));do
		PARTITIONS_CREATE_EVERY_DAY=$(date -d "${item} day ago" +"%Y%m%d")
		TIME_PARTITIONS=$(date -d "$(echo ${PARTITIONS_CREATE_EVERY_DAY} 23:59:59)" +%s)
		EXISTED_DATA_PARTITION_SQL[((index++))]="$(echo "PARTITION p${PARTITIONS_CREATE_EVERY_DAY}  VALUES LESS THAN (${TIME_PARTITIONS})")"
	done

	#future data
	for((item=1;item<=${HISTORY_DAYS};item++));do
		PARTITIONS_CREATE_EVERY_DAY=$(date +"%Y%m%d" --date="${item} days")
		TIME_PARTITIONS=$(date -d "$(echo ${PARTITIONS_CREATE_EVERY_DAY} 23:59:59)" +%s)
		EXISTED_DATA_PARTITION_SQL[((index++))]="$(echo "PARTITION p${PARTITIONS_CREATE_EVERY_DAY}  VALUES LESS THAN (${TIME_PARTITIONS})")"
	done
	
	for((item=0;item<=${HISTORY_DAYS};item++));do
		echo ${EXISTED_DATA_PARTITION_SQL[((item))]}
		if [ "${item}" == "0" ];then
			PARTITION_SQL=$(echo "${EXISTED_DATA_PARTITION_SQL[((item))]}")
		else
			PARTITION_SQL=$(echo "${PARTITION_SQL},${EXISTED_DATA_PARTITION_SQL[((item))]}")
		fi
	done;

	SQL2=$(echo "ALTER TABLE history PARTITION BY RANGE( clock ) (${PARTITION_SQL});")
	echo "=======================================begin do partition: ${TABLE_NAME}======================================"
	echo ${SQL2}
    RET2=$(${MYSQL_CMD} -e "${SQL2}")
	echo "=======================================end do partition: ${TABLE_NAME}======================================"

}
create_partitions_history_with_existed_data

RANGE Problems with partitioning ：

1. range Coverage issues ： When the value of the partition key in the inserted record is not in the range of the partition definition , Insert statements fail .

terms of settlement

1. Estimate the value of the partition key , Add new partitions in time

2. List Partition

List Partition is to establish a discrete value list to tell the database which partition a specific value belongs to . grammar ：

partition by list(exp)( //exp For column names or expressions 
        partition p0 values in (3,5)  // The value is 3 and 5 The in p0 Partition 
    )

example ：

mysql> create table emp1(
    -> id int not null,
    -> store_id int not null
    -> )
    -> partition by list(store_id)(
    ->   partition p0 values in (3,5),
    ->   partition p1 values in (2,6,7,9)
    -> );

Be careful If the value of the partition key corresponding to the inserted record is not in list In the value specified by the partition , Will fail to insert .

3. Hash Partition

Hash Partition is mainly used to disperse hot reading , Make sure that the data is distributed as evenly as possible in a predetermined number of partitions . MySQL Two kinds of support Hash Partition : routine Hash Zoning and linearity Hash Partition . A. routine Hash Partition : Using the modulus algorithm grammar ：

partition by hash(store_id) partitions 4;
 The sentence above , according to store_id Yes 4 modulus , Decide where records are stored .  such as store_id = 234 The record of ,MOD(234,4)=2, So it will be stored in the second partition .

routine Hash Advantages and disadvantages of partitioning advantage ： It can make the data as evenly distributed as possible . shortcoming ： It's not suitable for the changing needs of the partition . If I want to add two new partitions , Now there is 6 Zones , that MOD(234,6) The results are the same as before MOD(234,4) There's going to be inconsistencies in the results , So most of the data needs to be recalculated .

4. Key Partition

similar Hash Partition ,Hash Partitions allow user-defined expressions , but Key User defined expressions are not allowed for partitions .Hash Only integer partitions are supported , and Key Partition support in addition to Blob and text Other types of columns as partition keys . grammar

partition by key(exp) partitions 4;//exp Is a list of zero or more field names 
key When you partition ,exp Can be null , If it is empty , The primary key is used as the partition key by default , When there is no primary key , The non empty only key will be selected as the partition key .

The district is right NULL Value handling

1. range Partition ： NULL Values are treated as minimum values
2. List partition ：NULL The value must appear in the list , Otherwise it will not be accepted
3. Hash/Key partition ：NULL Values are treated as zero values

The impact of database engine on partition

Be careful ： Engine is innodb, stay innodb In order to succeed in partitioning technology in database engine, tables must be set as independent table spaces. Pay special attention to this .
Turn on innodb Database engine independent tablespace .
edit my.cnf increase innodb_file_per_table=1

[[email protected] test]# vi /etc/my.cnf
[mysqld]
innodb_file_per_table=1
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
max_connections=2000

Zone Management

Partition management includes the addition of partitions , Delete , And inquiry

• Add partitions

 Yes range/list Partition ：  alter table table_name add partition (partition p0 values ...(exp))
 Yes hash/key Partition ：    alter table table_name add partition partitions 8;   It's new 8 Zones 

• Delete partition

 Yes range/list Partition ：  alter table table_name drop partition p0; //p0 Name of the partition to delete 
 Yes hash/key Partition ：    alter table table_name coalesce partition 2; // Reduce the partition to 2 individual 

• Query partition

select partition_name,table_rows from information_schema.partitions where table_name="history";