Query efficiency increased by 10 times! Three optimization schemes to help you solve the deep paging problem of MySQL

Development often meets the need of paging query , But when you turn too many pages , Deep paging will occur , This leads to a sharp decline in query efficiency .

Is there any way , It can solve the problem of deep paging ？

This paper summarizes three optimization schemes , Query efficiency is directly improved 10 times , Let's study together .

1. Prepare the data

First create a user table , Only in create_time Index the fields ：

CREATE TABLE `user` (
  `id` int NOT NULL AUTO_INCREMENT COMMENT ' Primary key ',
  `name` varchar(255) DEFAULT NULL COMMENT ' full name ',
  `create_time` timestamp NULL DEFAULT NULL COMMENT ' Creation time ',
  PRIMARY KEY (`id`),
  KEY `idx_create_time` (`create_time`)
) ENGINE=InnoDB COMMENT=' User table ';

Then insert 100 10000 test data , Here you can use stored procedures ：

drop PROCEDURE IF EXISTS insertData;
DELIMITER $$
create procedure insertData()
begin
 declare i int default 1;
   while i <= 100000 do
         INSERT into user (name,create_time) VALUES (CONCAT("name",i), now());
         set i = i + 1; 
   end while; 
end $$

call insertData() $$

2. Verify the deep paging problem

each page 10 strip , When we look up the first page , fast ：

select * from user 
where create_time>'2022-07-03' 
limit 0,10;

In less than 0.01 It returns directly within seconds , So the execution time is not displayed .

When we turn to No 10000 Page time , Query efficiency drops sharply ：

select * from user 
where create_time>'2022-07-03' 
limit 100000,10;

Execution time becomes 0.16 second , The performance has decreased at least dozens of times .

Where is the time spent ？

1. Need to scan before 10 Data , Large amount of data , More time-consuming
2. create_time It's a non clustered index , You need to query the primary key first ID, Go back to the table again , Through primary key ID Query out all fields

Draw the back table query process ：

1. Through the first create_time Query the primary key ID

2. Through the primary key ID Query all fields in the table

Don't ask why B+ The structure of the tree is like this ？ Asking is the rule .

Then we will optimize for these two time-consuming reasons .

3. Optimized query

3.1 Use subquery

First, use the sub query to find out the qualified primary keys , Then use the primary key ID Make conditions to find out all fields .

select * from user 
where id in (
  select id from user 
  where create_time>'2022-07-03' 
  limit 100000,10
);

However, such a query will report an error , It is said that the use of limit.

We add a layer of sub query nesting , That's all right. ：

select * from user 
where id in (
 select id from (
    select id from user 
    where create_time>'2022-07-03' 
    limit 100000,10
 ) as t
);

The execution time is shortened to 0.05 second , Less 0.12 second , It is equivalent to that the query performance is improved 3 times .

Why use subquery to find qualified primary keys first ID, It can shorten the query time ？

We use it explain Just look at the implementation plan ：

explain select * from user 
where id in (
 select id from (
    select id from user 
    where create_time>'2022-07-03' 
    limit 100000,10
 ) as t
);

You can see Extra Column displays the... Used in the subquery Using index, It means that Overlay index , Therefore, sub queries do not need to be queried back to the table , Speed up query efficiency .

3.2 Use inner join Relational query

Treat the result of subquery as a temporary table , Then perform an association query with the original table .

select * from user 
inner join (
   select id from user 
    where create_time>'2022-07-03' 
    limit 100000,10
) as t on user.id=t.id;

Query performance is the same as using subqueries .

3.3 Use paging cursor （ recommend ）

The implementation method is ： When we look up the second page , Put the query results on the first page into the query conditions on the second page .

for example ： First, check the first page

select * from user 
where create_time>'2022-07-03' 
limit 10;

Then check the second page , Put the query results on the first page into the query conditions on the second page ：

select * from user 
where create_time>'2022-07-03' and id>10 
limit 10;

This is equivalent to querying the first page every time , There is no deep paging problem , Recommended .

Execution time is 0 second , The query performance has been directly improved by dozens of times .

Although such a query method is easy to use , But it brings another problem , Just can't jump to the specified number of pages , You can only turn down one page .

So this kind of query is only suitable for specific scenarios , For example, information APP Home page .

Internet APP Usually in the form of waterfall flow , For example, baidu home page 、 The front page of the headlines , All slide down and turn the page , There is no need to jump to the number of pages .

Don't believe it , You can look at it , This is the waterfall of headlines ：

The query result of the previous page is carried in the transmission parameter .

Response data , Returned the query criteria on the next page .

Therefore, the application scenario of this query method is quite wide , Use it quickly .

Summary of knowledge points ：