Summary of redis big key problem handling

Hello everyone , I've been busy with my work recently , The article went on for some time , Later, the pit will be filled slowly . today , Let's share a summary of our team members 《Redis Big key Problem handling summary 》, It can be said that the dry goods are full .

---

Redis Big key Problem handling summary

You can get ：

Catalog

In daily life Redis In use , We all met because key Too much data is stored, causing requests to be slow or even blocked , This is the time to check Redis The big key To optimize the business , The following provides a summary of some troubleshooting schemes , For reference only .

What size key Calculate big

Alibaba cloud Redis Best practices mention ：

reasonable Key in Value Byte size of , Less than recommended 10 KB. Too much Value Will cause data skew 、 hotspot Key、 Instance traffic or CPU Problems such as full performance , The performance impact of such problems should be avoided from the design source .

that value Bytes >10kb Can be used as a judgment Big key A reference value of .

How do you find big key

1、string Type can use the command --bigkeys

--bigkeys
yes redis The order that comes with you , To the whole Key scan , Statistics string,list,set,zset,hash The largest of these common data types key.string The type of statistics is value Bytes of ; in addition 4 Types of complex structures count the number of elements , Can not be seen intuitively value Bytes occupied , therefore --bigkeys
For analysis string Large of type key It is useful to , The type of complex structure also needs some third-party tools .（ Fewer elements , not always value not big ; There are many elements , Not necessarily. value Big ）

[email protected]:~# redis-cli -h 127.0.0.1 -p 6379 -a "password" --bigkeys 

--bigkeys
　 In order to scan Scan all by delaying calculation key, Therefore, there will be no blocking during execution redis, But there are a large number of instances keys when , The command takes a long time to execute , In this case, it is suggested that slave Scan up .

image

Mainly summary Part of the information , Analysis can be seen ：

This instance has a total of Yes 52992 individual key, Total occupied memory space ：bytes is 1470203;

list type Yes 15 individual key, maximal key Yes 153462 Elements ;list Type of key Account for the 00.03%, Average size 10232.67 Bytes.

hash   type   Yes 11485 individual key, maximal key There are three fields .

string type maximal key Take up memory space 157374 bytes.

set type maximal key Yes 10 Elements .

streams The type is an open source version 5.0 New data types supported ,pub/sub Enhanced version of type .

–-bigkeys
In fact, it is to find the largest of the types key, maximal key It doesn't have to be big key, maximal key Not more than 10kb Words , It means that there is no big key. But if a certain type has more big key (>10kb), Statistics only top1 the key, If you want to count all greater than 10kb Of key, Third party tools are required scanning rdb Persistent files .

2/ Not string Types are usually counted in the following two ways ：

• 2.1/ Redis 4.0 Later versions ： Support 了  memory Command view key Size

[email protected]:~# redis-cli -h 127.0.0.1 -p 6379 -a "password"
127.0.0.1:6379> MEMORY USAGE keyname1
(integer) 157481
127.0.0.1:6379> MEMORY USAGE keyname2
(integer) 312583

memory   The command counts an estimate . And –-bigkeys
 summary The statistics are slightly different .

• 2.2/ Rdbtools tool kit ：Rdbtools yes python Written A third-party open source tool , Used to resolve Redis Snapshot file . In addition to parsing rdb file , It also provides Count individual key Tools of size .

 #  Install as follows 
 git clone https://github.com/sripathikrishnan/redis-rdb-tools
 cd redis-rdb-tools sudo && python setup.py install

Rdbtools   Third party open source toolkit , from dump.rdb Snapshot file statistics （bgsave）, Will all > 10kb Of key Output to a csv file

[email protected]:~# rdb dump.rdb -c memory --bytes 10240 -f live_redis.csv 

How to delete big key

4.0 before string,list,set,hash   Large data with different data types key, The deletion method is different . There are generally two situations ：del Command to delete a single large key   and  del Batch deletion Big key. direct del Order a rude deletion key Easy to cause redis Thread blocking .4.0 In the past, elegant deletion was for different types Write the script , Split the list ,hash surface , Delete in batches .

4.0 After the version, the official version of Delete Big key With special optimization , Support lazy free function , Usually you can delete without developing scripts .

4.0 How to gracefully delete the big key Well ？

1/ Active deletion of large key

127.0.0.1:6379> UNLINK mykey

unlink The order is  del The asynchronous version of , from Lazyfree Mechanism realization .Lazyfree The principle of the mechanism is to delete only logically , hold key Release operation on bio (Background I/O) Lazy processing in separate child threads , Reduce delete size key Yes redis Main thread blocking , Effectively avoid large key Performance issues .unlink Even in batch deletion Big key when , It won't block the traffic .

2/ Passively delete large key

Passive deletion means Redis Self key Clear strategy , One Big key When it expires or is eliminated , How to be cleared , Will it cause congestion ？4.0 In the past, it was possible for automatic cleanup to block the main thread .

4.0 Later versions , Passive deletion policy is an optional configuration parameter , allow With Lazyfree The way to get rid of . But the parameter is off by default , The following parameters can be configured to enable .

lazyfree-lazy-expire on    #  Delete after expiration 
lazyfree-lazy-eviction on    #  The maximum memory is exceeded 
lazyfree-lazy-server-del on    #  The server is passively and lazily deleted 

summary

• Use Redis 4.0 Version above

• –-bigkeys
  、memory
  command and  Rdbtools Tools Analyze the problems of statistical examples Big key

• Big key problem , Actively delete unconditional use UNLINK asynchronous , Configure when deleting passively lazyfree Lazy deletion .

Last , Fundamentally solve the big problem key The problem should be avoided from the beginning key.

Reference documents

• elementary analysis Redis 4.0 The new features LazyFree

• Cloud database Redis Develop O & M specifications