Redis notes (13) - scan and keys search for specific prefix key fields (command format, usage examples, locating large keys)

1. keys

Redis Provides a simple violent instruction keys Used to list all the rules that satisfy a particular regular string key.

127.0.0.1:6379> keys *
(empty array)
127.0.0.1:6379> set wohu1104go1 1
OK
127.0.0.1:6379> set wohu1104go2 2
OK
127.0.0.1:6379> set wohu1104go3 3
OK
127.0.0.1:6379> set wohu1104go4 5
OK
127.0.0.1:6379> set wohu1104go5 5
OK
127.0.0.1:6379> set wohu1104python 1
OK
127.0.0.1:6379> set wohu1104python2 2
OK
127.0.0.1:6379> set wohu1104python3 3
OK
127.0.0.1:6379> set wohu1104python4 4
OK
127.0.0.1:6379> keys *
1) "wohu1104python4"
2) "wohu1104go3"
3) "wohu1104python3"
4) "wohu1104go1"
5) "wohu1104go4"
6) "wohu1104python2"
7) "wohu1104go2"
8) "wohu1104python"
9) "wohu1104go5"
127.0.0.1:6379> keys wohu1104go*
1) "wohu1104go3"
2) "wohu1104go1"
3) "wohu1104go4"
4) "wohu1104go2"
5) "wohu1104go5"
127.0.0.1:6379> keys wohu1104python*
1) "wohu1104python4"
2) "wohu1104python3"
3) "wohu1104python2"
4) "wohu1104python"
127.0.0.1:6379> 

This instruction is very simple to use , Just provide a simple regular string , But there are two obvious shortcoming .

1. No, offset、limit Parameters , Spit out all that meet the conditions at one time key, In case there are millions of instances key Meet the conditions , The output is difficult to analyze .
2. keys The algorithm is a traversal algorithm , Complexity is O(n), If there are more than ten million key, This command will lead to Redis Service carton , All reading and writing Redis All other instructions will be delayed or even over reported , because Redis It's a single threaded program , Execute all instructions in sequence , Other instructions must wait until the current keys After the command is executed, you can continue .

2. scan

2.1 command

scan Command is a cursor based iterator , After each call , Will return a new cursor to the user , The user needs to use this new cursor as the scan Cursor parameters for the command , To continue the previous iteration process .
​
scan Returns an array of two elements , The first element is a new cursor for the next iteration , The second element is an array , This array contains all the elements that are iterated . If the new cursor returns 0 Indicates that the iteration is over .
​
scan An instruction is a series of instructions , In addition to traversing all key outside , You can also traverse the specified container collection . such as

• zscan Traverse zset Collection elements
• hscan Traverse hash The elements of a dictionary
• sscan Traverse set Elements of the set .

Their principle is the same as scan It's going to be similar , because hash At the bottom is the dictionary ,set It's also a special hash( be-all value Point to the same element ),zset A dictionary is also used internally to store all the element contents .

scan The basic syntax of the command is as follows ：

SCAN cursor [MATCH pattern] [COUNT count]

• cursor - The cursor .
• pattern - Matching patterns .
• count - Specifies how many elements to return from the dataset , The default value is 10 .

The first time I traverse ,cursor The value is 0, Then the first integer value in the returned result will be used as the next traversal cursor. Go all the way back to cursor The value is 0 End of time .

2.2 characteristic

Redis In order to solve this keys Performance issues , It's in 2.8 The directive has been added to the version —— scan.scan comparison keys It has the following characteristics :

• The complexity is O(n), But it's done step by step with cursors , Will not block threads ;
• Provide limit Parameters , You can control the maximum number of results returned each time ,limit Just one. hint, More or less results can be returned ;
• Same as keys equally , It also provides pattern matching ;
• The server does not need to save state for cursors , The only state of a cursor is scan Cursor integer returned to the client ;
• The returned results may be repeated , The client needs to be de duplicated , This is very important ;
• If there is data modification during traversal , It is uncertain whether the modified data can be traversed ;
• Just because the result of a single return is empty doesn't mean the end of the traversal , It depends on whether the returned cursor value is zero ;

2.3 Use

Use the following code to Redis Add 1000 With the same prefix key

import redis

client = redis.Redis("127.0.0.1", 6379)
for i in range(1000):
    client.set("wohu{}".format(i), i)

Suppose we are looking for wohu520 This key, Then use the following command ,

127.0.0.1:6379> scan 0 match wohu52* count 100
1) "488"
2) 1) "wohu529"
127.0.0.1:6379> scan 488 match wohu52* count 100
1) "372"
2) 1) "wohu52"
127.0.0.1:6379> scan 372 match wohu52* count 100
1) "242"
2) 1) "wohu523"
   2) "wohu521"
   3) "wohu522"
127.0.0.1:6379> scan 242 match wohu52* count 100
1) "342"
2) (empty array)
127.0.0.1:6379> scan 342 match wohu52* count 100
1) "449"
2) (empty array)
127.0.0.1:6379> scan 449 match wohu52* count 100
1) "261"
2) 1) "wohu527"
127.0.0.1:6379> scan 261 match wohu52* count 100
1) "93"
2) 1) "wohu525"
127.0.0.1:6379> scan 93 match wohu52* count 100
1) "139"
2) 1) "wohu524"
   2) "wohu528"
127.0.0.1:6379> scan 139 match wohu52* count 100
1) "279"
2) (empty array)
127.0.0.1:6379> scan 279 match wohu52* count 100
1) "0"
2) 1) "wohu526"
   2) "wohu520"
127.0.0.1:6379> 

From the above process, we can see that although the limit yes 100, But there are only a few or no results returned . Because of this limit Not to limit the number of returned results , Instead, it limits the number of dictionary slots that the server can traverse at a single time ( About equal to ). If you will limit Set to 10, You will find that the return result is empty , But the cursor value is not zero , It means that the traversal is not over yet .

2.4 Storage structure

stay Redis All of the key It's all stored in a big dictionary , It's a one-dimensional array + Two dimensional linked list structure , The size of the first dimensional array is always 2^n(n>=0), Expand once, double the size of the array , That is to say n++.
​

​

scan The cursor returned by the instruction is the position index of the first dimension array , We call this location index slot (slot). If we do not consider the expansion and contraction of the dictionary , Directly by array subscript one by one traversal on the line .limit The parameter indicates the number of slots to traverse , The reason why the results returned may be more or less , Because not all slots will be linked to the list , Some slots may be empty , There are also some slots where the linked list may have more than one element . Every traversal will result in limit After pattern matching and filtering of all linked list elements on a number of slots , One time return to the client .

3. Big key location

Sometimes it's because the business people don't use it properly , stay Redis A large object will be formed in the instance , Like a big one hash, A big one zset It's all frequent . Such an object is right Redis The migration of cluster data brings a lot of problems , On memory allocation , If one key Too big , So when it needs to be expanded , A larger block of memory will be applied at one time , It's also going to lead to carton . If this is big key Be deleted , Memory will be recycled at one time , Once again, the Caton phenomenon will occur .
​
If you observe Redis Memory goes up and down , This is most likely because of the big key As a result of , At this time, you need to locate the specific one key, Further identify specific business sources , Then improve the relevant business code design .
​
To avoid alignment Redis Bring Caton , That's what we need scan Instructions , For each one scanned key, Use type Command acquisition key The type of , Then use the corresponding data structure size perhaps len Method to get its size , For each type , Keep the front of the size N The name is displayed as the scan result .
​
The above process needs to be scripted , More complicated , however Redis The authorities are already in redis-cli The command provides such a scanning function , We can use it directly .

redis-cli -h 127.0.0.1 -p 6379 –-bigkeys 

If you're worried that the order will go up a lot Redis Of ops Cause an online alarm , You can also add a sleep parameter .

redis-cli -h 127.0.0.1 -p 6379 –-bigkeys -i 0.1 

The above instruction is every 100 strip scan The command will sleep 0.1s,ops It's not going to lift so much , But the scan will take longer .
​

Reference resources ：
https://juejin.cn/book/6844733724618129422/section/6844733724710404110​