2. MySQL index optimization

Index optimization analysis

Performance degradation ,SQL slow 、 Long execution time 、 Long waiting time

• Excessive data —— Sub database and sub table
• Too many tables associated , Too much join——SQL Optimize
• Not making full use of the index —— Index building
• Server tuning and parameter settings —— adjustment my.cnf

Optimization means “ Index ” The quickest and most commonly used .

preheating — Common and general join Inquire about

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-xZtVJ9KX-1627788034664)(D:\QianFeng\ Blog \join Seven ways to link .png)]

among ：union Keywords when used , The fields of the two results are equal 、 Equal number of fields 、 The fields are in the same order .

union all and union The difference is that duplicate removal ,union It's going to be weightless .

select a.*,b.* from t_emp a 
left join t_dept b on a.deptid = b.id
where b.id is null
union
select a.*,b.* from t_dept b 
left join t_emp a on a.deptid = b.id
where a.id is null;
--  Inquire about a Table unique and b Table unique data 

mysql Single table bottleneck 500w

Index introduction

Indexes (Index) Help MySQL Data structure for efficient data acquisition .

The essence of index ： An index is a data structure

An index can be understood as " Quickly find data structures in order ".

The database system maintains a data structure that satisfies a specific search algorithm , These data structures reference data in some way , In this way, advanced search algorithms can be implemented on these data structures . These data structures are indexes .

Where the index is stored ： Generally speaking , The index itself is big , It's impossible to store everything in memory , therefore , Indexes are often stored on disk in the form of index files .

Advantages of index ：

• Improve the efficiency of data retrieval , Reduce the IO cost ;
• Sort data by index , Reduce the cost of sorting data , Reduce CPU Consumption of .

The disadvantages of indexing ：

• While improving the query speed , Reduces the speed of table updates . Because when updating the table ,MySQL Not only save table data , You also need to save the index column fields added each time in the index file
• The index is actually a table , The table holds the primary key and index fields , And points to the record of the entity table , So the index also takes up space .

The structure of the index ：

B_tree Balance tree

• Btree
  1. Non leaf nodes contain
     1. The data that the index points to
     2. The downward pointer
     3. Pointer to data
• B+tree
  • Non leaf nodes do not contain the data pointed to by the index

MySQL choice B+tree As index , Relatively limited memory ,B+tree It takes up more space than Btree One third smaller , Relative occurrence IO Fewer times , Less time .

Time complexity

The same problem can be solved by different algorithms , The quality of an algorithm will affect the efficiency of the algorithm and program . The purpose of algorithm analysis is to select the appropriate algorithm and improve the algorithm .

O(n) Time complexity The algorithm varies by orders of magnitude n The increasing complexity in the time dimension .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-MciZf6g7-1627788034670)(D:\QianFeng\ Blog \Mysql_ Learning record \image-20210615215759545.png)]

Clustered index and non clustered index

Cluster index is not a separate index type , It's a way of storing data .

“ Clustering ” Indicates that data rows and adjacent inter value clusters are stored together .

Only the primary key index is a clustered index , That is, the index arranged in order .

In addition to the primary key index , Other indexes are non clustered indexes , characteristic ： Non clustered indexes need to find all indexes .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-exstmr40-1627788034673)(image-20210615221121122.png)]

mysql Index classification

Single value index

​ That is, an index contains only a single column , A table can have multiple single-column indexes

​

create table 
customer(
    id int(10) unsigned auto_increment,
    customer_no varchar(200),
    customer_name varchar(200),
    primary key(id),
    key(customer_name)
    );
-- key(customer_name)  With the creation of the table , Create indexes at the same time 

unique index

​ The value of the index column must be unique , But you can have an empty value

primary key

​ After setting it as the primary key, the database will automatically create an index ,Innodb Index for clustering

Composite index

​ That is, an index contains multiple columns . Create an index for multiple fields

​ create index idx_age_deptid_name on t_emp(age,deptid,name)

​ The result of the query by matching the index is n Data , That is, if a composite index contains several fields, it will return the query results of several fields .

Basic grammar

• establish create [unique] index [indexName] on table_name(column)

  • Create a single value index

    Give table t_emp In the table name Field creation index ：

    create index idx_name on t_emp(name)

  • The creation of a unique index

    stay mysql in , With the creation of the primary key, the index will be automatically generated , Therefore, the unique index can be set for the unique fields in the data set of other fields in the table

    Create a unique index for the employee number field in the employee table

    create unique index idx_empno in temp(emp_no)

  • Composite index

• Delete drop index [indexName] on mytable;

• see show index from table_name\G

• Use alter command （ General use create That's enough ）

  --  There are four ways to add indexes to data tables 
  
  alter table tbl_name add primary key(column_list);
  --  This statement adds a primary key , This means that the index value must be unique and not empty 
  
  alter table tbl_name add unique index_name(column_list);
  --  The value of the index created by this statement must be unique （ except Null Outside ,Null There may be many times 
  
  alter table tbl_name add index index_name(column_list);
  --  Add a normal index , Index values can appear multiple times 
  
  alter table tbl_name add fulltext index_name(column_list);
  --  This statement specifies that the index is fulltext, For full-text indexing 
  

When to create an index

• The primary key automatically creates a unique index ;;
• Fields that are frequently used as query criteria should be indexed ;
• Fields associated with other tables in the query , Index foreign key relationship ;
• Single key / The choice of Composite Index , Composite index is more cost-effective ;
• Fields sorted in the query , If the sorting field is accessed through the index, the sorting speed will be greatly improved ;
• Statistics or grouping fields in a query ,
  • group by Compare with order by More damaging to performance , because group by Contained in the order by Sort first, then group .

In those cases, it is not allowed to create an index

• There are too few records
• A table or field that is frequently added, deleted, or queried
• where There is no need to create an index for fields that are not used in the condition
• Those with poor filtering are not suitable for index creation

Measure whether the index should be built , How to build an index “ Ruler ”

Explain Namely mysql Provided to users “ Ruler ”

One 、Explain What is it? （ View execution plan ）

Implementation plan ： stay MySQL In the logical architecture optimizer( Optimizer ) Adjust without changing the query results sql Execution order , Generate execution plan .

Use EXPLAIN Keyword can simulate optimizer execution SQL Query statement , So they know MySQL How to deal with you SQL Of the statement . Analyze the performance bottleneck of your query statement or table structure .

Two 、Explain Can do ？

• Read order of tables

• Which indexes can be used

• Operation type of data read operation

• Which indexes are actually used

• References between tables

• How many rows per table are physically queried

3、 ... and 、Explain How to use it?

EXPLAIN sql Query statement 

Explain sql sentence Return to right sql Statement analysis .

Information contained in the implementation plan （ pivotal ）：

1. id

   • select The serial number of the query , Contains a set of numbers , Represents execution in a query select The order of clauses or operation tables
   • Three id situation ：
     • id Same number ： The order of execution is from top to bottom （ Analysis table shown ）
     • id inequality ： If it's a subquery ,id The serial number of will increase ,id The higher the priority, the higher , The first to be executed ;
     • id In the column id There are both similarities and differences ： According to the first id Different , Press again id Prioritize the same
   • concerns ：id Number, each number , Represents an independent query , One sql The fewer query times, the better .（ The same number represents a query ）

2. type

   Shows what type of query is used .

   • From the best to the worst ：

     system>const>eq_ref>ref>range>index>ALL

     • all： Full table scan 、 Extremely inefficient （ When type This keyword appears in the field , It can be indexed ）
     • index： Overlay index , appear index when sql Index is used but not filtered by index , Generally, the overlay index is used or the index is used for sorting and grouping ;（ It needs to be optimized ）
     • range： Retrieve only rows in the given range , Use an index to select rows .key Column shows which index is used . It's usually in your where In the sentence between、<、>、in And so on , This range scan index scan is better than full table scan , Because it just needs to start at a certain point in the index , And end at another point , It does not involve scanning all indexes

Thinking questions ： How to fill a lot of data into the bid

1. Insert multiple rows of data in one statement

2. Advantages and disadvantages of index

   1. advantage ： Query sorting is fast
   2. shortcoming ： All writes slow down

   So delete all indexes except the primary key index in the table , It can speed up the insertion

3. Turn off automatic commit of transactions , Commit transactions manually , Insert multiple pieces of data at one time

4. Multi threaded data insertion , Achieve optimization .

Query optimization

Batch data script

• Insert... Into the table 50w data

  1. Build table

  CREATE TABEL 'dept'(
  	'id' INT(11) NOT NULL AUTO_INCREMENT,
      'deptName' VARCHAR(30) DEFAULT NULL,
      'address' VARCHAR(30) DEFAULT NULL,
      'ceo' INT NULL,
      PRIMARY KEY('id')
  )ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
  
  CREATE TABLE 'emp'(
  	'id' INT(11) NOT NULL AUTO_INCREMENT,
      'empno' INT NOT NULL,
      'name' VARCHAR(20) DEFAULT NULL,
      'age' INT(3) DEFAULT NULL,
      'deptid' INT(11) DEFAULT NULL,
      PRIMARY KEY('id')
  )ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
  
  

  2. Set parameters log_bin_trust_function_creators

     Check whether the parameter is on ：SHOW VARTABLES LIKE 'log_bin_trust_function_creators'

     log_bin ： yes mysql Binary log , Realization MySQL Used in master-slave replication ,MySQL The default is off .

     Open the parameter ：SET GLOBAL log_bin_trust_function_creators=1

     It's about “global”, If you don't fill in , Then this parameter is only valid in the current window ; If you add , All windows as a whole are valid .

  3. Create a function , Make sure every data is different

     #  Randomly generate strings 
     DELIMITER $$
     # delimiter  Modify the Terminator 
     CREATE FUNCTION rand_string(n INT) RETURNS VARCHAR(255)
     BEGIN
     DECLARE chars_str VARCHAR(100) DEFAULT
     'abcdefghijklmnopqrstuvwxyzABCDEFGHIGKLMNOPQRSTUVWXYZ';
     DECLARE return_str VARCHAR(255) DEFAULT;
     DECLARE i INT DEFAULT 0;
     WHILE i<N DO
     SET return_str=CONCAT(return_str,SUBSTRING(chars_str,FLOOR(1+RAND()*52),1));
     SET i=i+1;
     END WHILE;
     RETURN return_str;
     END$$
     
     #  Add to delete 
     # drop function rand_string;
     
     #  Randomly generate department number 
     DELIMITER $$
     CREATE FUNCTION rand_num(from_num INT,to_num INT) RETURNS INT(11)
     BEGIN
     DECLARE i INT DEFAULT 0;
     SET i=FLOOR(from_num+RAND()*(to_num - from_num + 1));
     RETURN I;
     END$$
     #  If you want to delete 
     # drop function rand_num;
     
     

     delimiter Modify the Terminator , Need to use MySQL Programming functions , So modify the terminator , To facilitate MySQL Programming .

  4. Create stored procedure

     Batch generation of data 、 derivative 、 Data operation

     • To create emp A stored procedure that inserts data into a table

       DELIMITER $$
       CREATE PROCEDURE insert_emp(START INT,max_NUM INT)
       BEGIN
       DECLARE i INT DEFAULT 0;
       #set autocommit = 0  hold autocommit Set to 0
       SEET autocommit=0
       REPEAT
       SET i=i+1;
       INSERT INTO emp(empno,NAME,age,deptid) 
       VALUES((START+i),
              rand_string(6),
              rand_num(30,50),
              rand_num(1,10000)       
             );
       UNTIL i = max_num
       END REPEAT;
       COMMIT;
       END$$
       

       PROCEDURE stored procedure

       SEET autocommit=0 Automatic submission =0, Will automatically submit closed

     • To create dept A stored procedure that inserts data into a standard

       Similarly, write the stored procedure of the Department table

       #  Execute stored procedures , towards dept  Add random data to the table 
       DELIMITER $$
       CREATE PROCEDURE 'insert_dept'(max_num INT)
       BEGIN
       DECLARE i INT DEFAUL 0;
       SET autocommit=0;
       REPEAT
       SET i=i+1;
       INSERT INTO dept(deptname,address,ceo)
       VALUES(
           rand_string(8),
           rand_string(10),
           rand_num(1,500000)
       );
       UNTIL i=max_num
       END REPEAT;
       COMMIT;
       END$$
       

  5. Calling stored procedure

     #  Execute stored procedures , towards dept Add 10000 pieces of data to the table 
     DELIMITER ;
     CALL inster_dept(10000);
     
     
     #  towards emp Add... To the table 50 Ten thousand data 
     CALL inster_emp(100000,50000);
     

It has been mentioned before , By creating an index, you can more quickly complete the query operation from the big data , However, in our work, we will focus on one sql Create multiple indexes , So there's a problem , Due to the existence of multiple indexes , We have to delete the index , To ensure the next sql The normal execution of .

Delete multiple indexes

Before deleting the index , We must first know which indexes need to be deleted , And then delete it .

1. Query index name

   SHOW INDEX FROM table_name, but show Just show , Cannot be extracted from .

   stay MySQL There is a table in the database to store index information , stay information_schema Metadata （ Data that describes data ） Save in Library ,STATISTICS Store... In the statistics table .STATISTICS The table stores the index name INDEX_NAME Table name corresponding to index TABLE_NAME And the name of the library TABLE_SCHEMA、 The sequence of indexes SEQ_IN_INDEX Etc .

   notes ： Be careful when deleting an index , The primary key index cannot be deleted ; For composite index delete , The index sequence is 1 The can

   As mentioned above , We can find that we want to delete t_emp Statement for index of table ：

   SELECT index_name FROM information_schema.STATISTICS WHERE table_name='t_emp' AND table_shema='mydb' AND index_name <> 'PRIMARY' AND seq_in_index = 1;

2. Take out the index name - character string

   Extract the index found in the first step through the cursor object

   CURSOR  The cursor 
   FETCH xxx INTO xxx
   

3. Concatenate index names drop The statement is a string type How to convert a string to sql

   precompile prepare precompile xxx EXECUTE

#  A stored procedure for batch index deletion 
DELIMITER $$
CREATE PROCEDURE 'proc_drop_index'(dbname VARCHAR(200),tablename VARCHAR(200))
BEGIN
	DECLARE done INT DEFAULT 0;
	DECLARE ct INT DEFAULT 0;
	DECLARE _index VARCHAR(200) DEFAULT *;
	DECLARE _cur CURSOR FOR SELECT index_name FROM  information_schema.STATISTICS WHERE table_name='t_emp' AND table_shema='mydb' AND index_name <> 'PRIMARY' AND seq_in_index = 1;
    DECLARE CONTINUE HANDLER FOR NOT FOUND set done=2;
    OPEN _cur;
    FETCH _cre INTO _index;
    WHILE _index<>"DO
    	SET @str=CONCAT("drop index",_index,"on",tablename);
    	PREPARE sql_str FROM @str;
    	EXECUTE sql_str;
    	DEALLOCATE PREPARE sql_str;
    	SET _index=";
    	FETCH _cur INTO _index;
	END WHILE;
CLOSE _cur;
END$$

#  Execute stored procedures 
CALL proc_drop_index("dbname","tablename");