[2022 dark horse programmer] SQL optimization

【 Black horse programmer 】SQL Optimize notes

Insert data optimization

If we need to insert multiple records into the database table at one time , It can be optimized from the following three aspects .

• Bulk insert data
• Manually control transactions
• Insert primary key in order , Performance is better than out of order insertion

Mass insert data

If you need to insert a large amount of data at one time ( such as : Millions of records ), Use insert Statement insertion performance is low , You can use MySQL Database provides load Instructions Insert . The operation is as follows ：

1. First, import the prepared data into root Under the table of contents , Create a new one itheima The database of
   

2. When the client connects to the server , Add parameters -–local-infile

   mysql –-local-infile -u root -
   

3. Set global parameters local_infile by 1, Turn on the switch to import data from the local load file

   set global local_infile = 1;
   

   You can see that local_infile Have been to 1

   

4. Create a table

   CREATE TABLE `tb_user` (
     `id` INT(11) NOT NULL AUTO_INCREMENT,
     `username` VARCHAR(50) NOT NULL,
     `password` VARCHAR(50) NOT NULL,
     `name` VARCHAR(20) NOT NULL,
     `birthday` DATE DEFAULT NULL,
     `sex` CHAR(1) DEFAULT NULL,
     PRIMARY KEY (`id`),
     UNIQUE KEY `unique_user_username` (`username`)
   ) ENGINE=INNODB DEFAULT CHARSET=utf8 ;
   

5. perform load The instruction will prepare the data , Load into table structure

   load data local infile '/root/sql1.log' into table tb_user fields terminated by ',' lines terminated by '\n' ;
   

6. You can see the insertion time

   

   stay load when , The performance of primary key sequential insertion is better than that of out of order insertion

summary

• Bulk insert data

  Insert into tb_test values(1,'Tom'),(2,'Cat'),(3,'Jerry');
  

• Manually control transactions

  start transaction;
  insert into tb_test values(1,'Tom'),(2,'Cat'),(3,'Jerry');
  insert into tb_test values(4,'Tom'),(5,'Cat'),(6,'Jerry');
  insert into tb_test values(7,'Tom'),(8,'Cat'),(9,'Jerry');
  commit;
  

• Insert primary key in order , Performance is better than out of order insertion

   The primary key is inserted out of order  : 8 1 9 21 88 2 4 15 89 5 7 3
   Insert primary key in order  : 1 2 3 4 5 7 8 9 15 21 88 89
  

• Large scale data , have access to load

Primary key optimization

How data is organized

stay InnoDB In the storage engine , Table data is organized and stored according to the primary key sequence , The tables in this way are called Index organization table (index organized table IOT).

Row data , Are stored on the leaf node of the clustered index

stay InnoDB In the engine , Data lines are recorded in logical structures page On page , And the size of each page is fixed , Default 16K. That means , The rows stored in a page are also limited , If the inserted data row row The size of is larger than the maximum storage capacity of the page , Then it will be stored in the next page , Pages are connected by pointers .

Page splitting

The page can be empty , You can fill half , You can also fill in 100%. Each page contains 2-N Row data ( If a row of data is too large , Line overflow ), Arrange according to the primary key .

A. Primary key sequence insertion effect

①. Request page from disk , Primary key sequential interpolation

②. The first page is not full , Continue to insert... On the first page

③. When the first one is full , Write the second page , Pages are connected by pointers

④. When the second page is full , Then write to the third page

B. The primary key is inserted out of order

①. Join in 1#,2# The pages are full , The data shown in the figure is stored

②. Insert again at this time id by 50 The record of , Let's see what happens

Will not open a page again , Insert into new page ,** because , The leaf nodes of the index structure are in order . In order , It should be stored in 47 after .** however 47 Where 1# page , It's full , Can't store 50 The corresponding data . Then a new page will be opened 3#. But it will not directly 50 Deposit in 3# page , It's about putting 1# The data in the second half of the page , Move to 3# page , And then in 3# page , Insert 50.

mobile data , And insert id by 50 After the data of , So at this time , The data order between these three pages is problematic . 1# Next page of , Should be 3#, 3# The next page is 2#. therefore , here , Need to reset the linked list pointer .

The above phenomenon , be called “ Page splitting ”, It is a performance consuming operation .

Page merge

When we delete existing data , The specific effects are as follows : When deleting a row of records , In fact, the record has not been physically deleted , Just the record is marked （flaged） To delete And its space becomes allowed to be used by other record declarations .

When we continue to delete 2# Data record of , When the record deleted in the page reaches MERGE_THRESHOLD（ The default is... Of the page 50%）,InnoDB Will start looking for the closest page （ front Or after ） See if you can merge two pages to optimize space usage .

Delete data , And merge the pages , Insert new data again 21, And just insert 3# page

The phenomenon of merging pages that occurs in this , It's called “ Page merge ”.

Knowledge tips ： MERGE_THRESHOLD： The threshold for merging pages , You can set it yourself , Specify... When creating a table or index .

Index design principles

• Meet business needs , Try to reduce the length of the primary key .
• When inserting data , Try to insert in order , Choose to use AUTO_INCREMENT Since the primary key .
• Try not to use UUID Make primary keys or other natural primary keys , Such as ID card No .
• During business operation , Avoid modifying the primary key .

order by Optimize

MySQL Sort , There are two ways ：

1. Using filesort : Through table index or full table scan , Read the data lines that meet the conditions , Then in the sort buffer sort buffer Complete the sorting operation in , All sorts that do not return sorting results directly through index are called FileSort Sort .
2. Using index : Through the sequential scan of the ordered index, the ordered data can be returned directly , This is the case using index, No need for extra sorting , High operating efficiency .

For the above two sorting methods ,Using index High performance , and Using filesort Low performance , When we optimize sorting operations , Try to optimize for Using index.

test

1. Put the previous test , by tb_user Some indexes established by the table are deleted directly

   drop index idx_user_phone on tb_user;
   drop index idx_user_phone_name on tb_user;
   drop index idx_user_name on tb_user;
   

   

2. Perform sorting SQL

   explain select id,age,phone from tb_user order by age ;
   

   
   ​ because age, phone There's no index , So when sorting again at this time , appear Using filesort, Low sorting performance .

3. Index

   create index idx_user_age_phone_aa on tb_user(age,phone);
   

4. After index creation , according to age, phone Sort in ascending order

   explain select id,age,phone from tb_user order by age
   

   

   explain select id,age,phone from tb_user order by age , phone;
   

   

​ After indexing , Sort again , From the original Using filesort, Change into Using index, The performance is relatively high .

5. After index creation , according to age, phone Sort in descending order
   

6. according to phone,age Sort in ascending order ,phone before ,age After .
   
   ​ Sorting time , Also need to satisfy the leftmost prefix rule , Otherwise, there will be filesort. Because when creating an index , age Is the first Field ,phone It's the second field , therefore Sorting time , We should follow this order , Otherwise, it will appear Using filesort.

7. according to age, phone In descending order, in ascending order , A descending order

   explain select id,age,phone from tb_user order by age asc , phone desc ;
   

   
   ​ Because when you create an index , If the order is not specified , By default, they are sorted in ascending order , And when you query , An ascending order , A descending order , here Will appear Using filesort.
   ​ In order to solve the above problems , We can create an index , In this joint index age Ascending sort ,phone Reverse sort .

8. create index idx_user_age_phone_ad on tb_user(age asc ,phone desc);
   

   

9. Then perform the following again SQL

   explain select id,age,phone from tb_user order by age asc , phone desc ;
   

   Support mysql8.0 edition

   

order by Optimize the rules

1. Establish the appropriate index according to the sorting field , When sorting multiple fields , Also follow the leftmost prefix rule .
2. Try to use overlay index .
3. Multi field sorting , One ascending, one descending , At this time, you need to pay attention to the rules when creating the federated index （ASC/DESC）.
4. If inevitable filesort, When sorting large amounts of data , You can appropriately increase the size of the sorting buffer sort_buffer_size( Default 256k).

group by Optimize

Execute without an index group by

Delete all indexes

drop index idx_user_pro_age_sta on tb_user;
drop index idx_email_5 on tb_user;
drop index idx_user_age_phone_aa on tb_user;
drop index idx_user_age_phone_ad on tb_user;

Execute query statement

explain select profession , count(*) from tb_user group by profession ;

then , We are aiming at profession , age, status Create a federated index .

create index idx_user_pro_age_sta on tb_user(profession , age , status)：

And then execute ：

explain select profession , count(*) from tb_user group by profession 

Only age Grouping

explain select  age, count(*) from tb_user group by age;

We found that , If only based on age grouping , Will appear Using temporary ; And if it is according to profession,age Two fields are grouped at the same time , Will not appear Using temporary. The reason is that for grouping operations , In the union index , It also conforms to the leftmost prefix rule .

Grouping optimization rules

therefore , In grouping operations , We need to optimize through the following two points , To improve performance ：

1. During grouping operation , Efficiency can be improved by indexing .
2. When grouping , The use of index also satisfies the leftmost prefix rule .

limit Optimize

When the amount of data is large , If carried out limit Paging query , In the query , The later , The lower the efficiency of paging query .

limit Comparison of paging query time

Through the test, we will see , The later , The lower the efficiency of paging query , That's the problem with paging queries .

Optimization idea : General paging query , Better performance can be achieved by creating indexes , You can add subqueries by overriding the index Query form is optimized .

Search without index

select * from  tb_user limit 999000,10;

Use the index to search

mysql> select id from tb_user order by id limit 999000,10;

Use overlay query + Socket sub query to search

 explain select * from tb_user t , (select id from tb_user order by id limit 999000,10) a where t.id = a.id;

count Optimize

select count(*) from tb_user 

If you want to improve significantly InnoDB Tabular count efficiency , The main optimization ideas ： Count yourself ( With the aid of redis Such a number According to the database , But if it's conditional count It's more troublesome again ).

count usage

count() It's an aggregate function , For the returned result set , Judge line by line , If count The argument to the function is not NULL, Add... To the cumulative value 1, Otherwise, we will not add , Finally return the cumulative value .

usage ：count（*）、count（ Primary key ）、count（ Field ）、count（ Numbers ）

In order of efficiency ,count( Field ) < count( Primary key id) < count(1) ≈ count(), So try to Amount of use count().

update Optimize

update course set name = 'javaEE' where id = 1 ;

When we delete SQL When the sentence is , Will lock id by 1 Data in this row , Then after the transaction is committed , Row lock release .

But when we are performing the following SQL when .

update course set name = 'SpringBoot' where name = 'PHP' ;

When we start multiple transactions , In performing the above SQL when , We found that row locks were upgraded to table locks . This leads to the update Statement performance Greatly reduced . because name Field has no index .

InnoDB The row lock of is the lock added for the index , It's not a lock on records , And the index cannot be invalidated , Otherwise, it will be upgraded from row lock to table lock .

nt( Primary key id) < count(1) ≈ count(), So try to Amount of use count().

update Optimize

update course set name = 'javaEE' where id = 1 ;

When we delete SQL When the sentence is , Will lock id by 1 Data in this row , Then after the transaction is committed , Row lock release .

But when we are performing the following SQL when .

update course set name = 'SpringBoot' where name = 'PHP' ;

When we start multiple transactions , In performing the above SQL when , We found that row locks were upgraded to table locks . This leads to the update Statement performance Greatly reduced . because name Field has no index .

InnoDB The row lock of is the lock added for the index , It's not a lock on records , And the index cannot be invalidated , Otherwise, it will be upgraded from row lock to table lock .

So go ahead update When , Try to use the index to judge .