Redis (12) -- redis server

1, Preface

Redis The server is responsible for establishing network connections with multiple clients , Handle command requests sent by clients , Save the data generated by the client executing the command in the database . And maintain the operation of the server itself through resource management .
Other knowledge points involved in this article ：
Database and expiration key
RDB Persistence
AOF Persistence
Redis client

2, The execution of a command request

From sending a command to getting a reply , The client and server need to complete a series of operations , Suppose the client executes the following command ：

redis> SET KEY VALUE
OK

Send... From client SET KEY VALUE From command to reply , Both the client and the server need to perform the following operations ：

1. The client sends a command request to the server SET KEY VALUE
2. The server receives and processes the command request from the client SET KEY VALUE, Set in the database , And generate a command reply OK
3. The server replies the command to OK Send to client
4. The client receives the command reply returned by the server OK, And print it to the user for viewing

2.1, Send command request

When the user types a command into the client , The client will convert this command to protocol format , Then by connecting to the server's socket , Send the command request in protocol format to the server . The schematic diagram is shown below ：

for example , Suppose the user typed SET KEY VALUE After the command

The client will convert this command into a protocol ：

*3\r\n$3\r\nSET\r\n$3\r\nKEY\r\n$5\r\nVALUE\r\n

Then send this paragraph to the server

2.2, Read command request

When the connection socket between the client and the server becomes readable due to the command input of the client , The server will call the command request processor to perform the following operations ：

1. Read command request in protocol format in socket , And save it to the input buffer of client state
2. Parse the command request of the input buffer , Extract the command parameters contained in the command request 、 The number of command parameters , And save them to the client state argv Properties and argc attribute
3. Call the command executor , Carry out orders

for example SET KEY VALUE After the command request in the corresponding protocol format is parsed , In the client state, it will be stored like this ：

And save the analysis results to argv Properties and argc attribute ：

Next, the server will call the command executor to complete the remaining steps required to execute the command ：

2.3, Command actuators

2.3.1, Find command implementation

The first thing the command executor has to do is based on the state of the client argv[0] Parameters （ In the example above , This parameter is SET）, In the command table （command table） Find the command specified by the parameter , And save the found command to the client state cmd Attributes inside （ If not found , Will cmd Attribute points to NULL）

The command list is a dictionary , The keys in the dictionary are command names , such as set、get、del wait ; The value of dictionaries is one by one redisCommand structure , Every redisCommand The structure records a Redis Command implementation information , The following table records the types and functions of the main attributes of this structure ：

Property name	type	effect
name	char *	The name of the order , such as set
proc	redisCommandProc *	A function pointer , Point to the implementation function of the command , such as setCommand.redisCommandProc Type defined as ：typedef void redisCommandProc(redisClient *c) 【 This means that you need to pass in a client state as a parameter 】
arity	int	The number of command parameters , Used to check whether the format of the command request is correct . If this value is negative , Indicates that the number of parameters is greater than or equal to N【 It is worth noting that ： The name of the command itself is also a parameter 】
sflags	char *	Identification value in string form , This value records the properties of the command , For example, is this command a read command or a write command , Whether this command is allowed to be used when loading data , Whether this command is allowed in Lua Use in scripts, etc
flags	int	Yes sflags The binary identification obtained from the analysis of the identification , The program automatically generates . The server uses flags Property instead of sflags attribute , Because the binary check can pass &、^、~ Wait to finish
calls	long long	How many times has the server executed this command
milliseconds	long long	The total time taken by the server to execute this command

Here are sflags The identification value that can be used by the property ：

identification	significance	Commands with this logo
w	This is a write command , The database may be modified	SET、RPUSH、DEL etc.
r	This is a read-only command , The database will not be modified	GET、STRLEN、EXISTS etc.
m	This command may take up a lot of memory , Before execution, you need to check the memory usage of the server , If memory is scarce, it is forbidden to execute	SET、APPEND、RPUSH、LPUSH、SADD、SINTERSTORE etc.
a	This is a management order	SAVE、BGSAVE、SHUTDOWN etc.
p	This is a command about publish and subscribe function	PUBLISH、SUBSCRIBE、PUBSUB etc.
s	This command cannot be used in Lua Use in script	BRPOP、BLPOP、BRPOPLPUSH、SPOP etc.
R	This is a random order , For the same dataset and the same parameters , The results returned by the command may be different	SPOP、SRANDMEMBER、SSCAN、RANDOMKEY etc.
S	When in Lua When using this command in a script , Sort the output of this command once , Make the results of the command orderly	SINTER、SUNION、SDIFF、SMEMBERS、KEYS etc.
l	This command can be used when the server is loading data	INFO、SHUTDOWN、PUBLISH etc.
t	This is a command that allows the slave server to use when it has expired data	SLAVEOF、PING、INFO etc.
M	This command is on the monitor （monitor） It will not be propagated automatically in mode	EXEC

Next, let's say SET and GET An example of an order , Exhibition redisCommand structure ：

• SET The name of the command is set; The implementation function is redisCommand; The number of command parameters is -3, It means that three or more parameters can be accepted ; The command ID is wm, Express SET The command is a write command , And you need to check the memory usage of the server before execution , Because this command may take up a lot of memory
• GET The name of the command is get; The implementation function is getCommand; The number of command parameters is 2, It means that the command accepts only two parameters ; The identification of the command is r, Indicates that this is a read-only command

Used as a program argv[0] As input, go to the command table to find （ For example, find set command ）, The command table will return set Corresponding redisCommand structure . Client state cmd The pointer will point to this redisCommand structure

It is worth mentioning that ： The search in the command table uses a case independent Algorithm , So as long as the command name is correct , You can find the corresponding redisCommand structure . This is the same. Redis The reason why the command input of the client is not case sensitive

2.3.2, Perform preliminary operations

Up to this point , The server has satisfied several conditions required to execute the command ：

• Command implementation parameters （ Stored in client state cmd attribute ）
• Parameters （ Stored in client state argv attribute ）
• Number of parameters （ Stored in client state argc attribute ）

But before the command is actually executed , The program also needs some preparatory operations （ The operations listed below are in stand-alone mode , When the server executes commands in replication or cluster mode , There will be more preparatory operations ）, This ensures that the command is correct 、 Smooth implementation . These operations are ：

• Check client status cmd Whether the attribute points to NULL, If so , Description the command name entered by the user cannot find the corresponding implementation function , It means that the command input is wrong , The server will not perform the next steps , And an error will be returned to the client
• According to client cmd Property points to redisCommand Structural arity attribute , Check whether the number of parameters given by the command request is correct , If not , Do not follow the next steps , And return an error to the client
• Check whether the client is authenticated , Unauthenticated clients can only execute AUTH command , If an unauthenticated client attempts to execute another command , The server will return an error to the client
• If the server is turned on maxmemory function , So before executing the command , Check the memory usage of the server first , And reclaim memory when necessary , So that the next command can be executed smoothly . If memory recall fails , Do not follow the next steps , And return an error to the client
• If the server last executed BGSAVE An error occurred when , And the server is turned on stop-writes-on-bgsave-error function , And the command that the server will execute is a write command , Then the server will refuse to execute this command and return an error to the client
• If the client is currently using SUBSCRIBE Command subscription channel , Or in use PSUBSCRIBE Command subscription mode , Then the server will only execute the SUBSCRIBE、PSUBSCRIBE、UNSUBSCRIBE、PUNSUBSCRIBE Four orders , Other commands will be rejected by the server
• If the server is loading data , Then only with l The marked command will be executed , Other orders will be rejected
• If the server Lua The script timed out and entered a blocked state , Then the server will only execute the SHUTDOWN nosave Command and SCRIPT KILL command , Other commands will be rejected by the server
• If the client is executing a transaction , Then the server will only execute the EXEC、DISCARD、MULTI、WATCH command , Other commands are put into the transaction queue
• If the server turns on the monitor function , Then the server will send information such as the command and parameters to be executed to the monitor

After completing the above preparatory operations , The server can actually execute commands

2.3.3, Call the implementation function of the command

In the previous operation , The server has the command to be executed 、 Parameters 、 The number of parameters is ready , When the server decides to execute the command , Just execute the following statement ：

//client  Is a pointer to the client state 
client->cmd->proc(client);
// As mentioned above set Command as an example , In fact, it is equivalent to executing ：setCommand(client)

The called command implementation function performs the specified operation , And generate corresponding command reply , These command replies are saved in the output buffer of the client , Then the implementation function replies to the processor for the socket Association command of the client , Used to return the command reply to the client

2.3.4, Carry out follow-up work

After executing the function , The server still needs to perform some follow-up work ：

• If the server has enabled the slow query log function , Then the slow query log will determine whether it is necessary to add a new slow query log for the command just executed
• According to the execution time of the command , Update the redisCommand Structural milliseconds attribute , And will redisCommand Structural calls The value of the counter is increased by one
• If the server is turned on AOF Persistence function , that AOF The persistence module will write the command request just executed to AOF buffer
• If there are other buffers copying the current server , Then the server will transmit the just executed command to all slave servers

When the above operations are completed , The server's execution of the current command is over , You can take it from the file event handler and process the next command request

2.4, Send command reply to client

The previously mentioned command implementation function will save the command reply to the output buffer of the client , And for the client socket associated command reply processor , When the client socket becomes writable , The server will execute the command to reply to the processor , Send the command reply saved in the client output buffer to the client

When the command reply is sent , The reply processor will empty the output buffer of the client , Be prepared to process the next command request

2.5, The client receives and prints the command reply

When the client receives the command reply in protocol format , Will convert these replies into a human readable format , And print it to the user ：

3,serverCron function

Redis Of serverCron Function every 100 Execute in milliseconds , This function is responsible for managing the resources of the server , And keep the server running well

3.1, Update server time cache

Redis There are many functions in the server that need to get the current time of the system , Each time you get the current time of the system, you need to execute a system call , In order to reduce the number of system calls , In server state unixtime Properties and mstime Property is used as a cache of the current time ：

struct redisServer{
    
    
    //...
    
    // Save the second accuracy of the system's current UNIX Time stamp 
    time_t unixtime;
    
    // The current system with millisecond precision is saved UNIX Time stamp 
    long long mstime;
    
    //...
    
};

because serverCron The function defaults to every 100 Millisecond frequency update unixtime Properties and mstime attribute , Therefore, the time accuracy of these two attribute records is not high . So only print the log 、 Update the server's LRU The clock 、 Decide whether to implement persistence 、 These two attributes are only used when calculating the online time of the server and other operations that do not require high time accuracy . For setting key expiration time 、 Adding slow query logs requires high-precision work , The server will make a system call to get the most accurate current time .

3.2, to update LRU The clock

In server state lrulock Property of the server LRU The clock , This attribute is the same as that described above unixtime attribute 、mstime Same property , Are a kind of server time cache ,serverCron The function defaults to every 10 Update every second lrulock The value of the property , Because the value of this clock is not real-time , So it is calculated according to this attribute LRU Time is actually just an estimate ：

struct redisServer{
    
    
    //...
    
    // The default for each 10 Clock cache that is updated every second 
    // Used to calculate the idle time of the key 
    unsigned lrulock:22;
    
    //...
    
};

Every Redis Every object will have one lru attribute , This lru Property holds the time when the object was last accessed by the command ：

typedef struct redisObject{
    
    
    //...
    
    unsigned lru:22;
    
    //...
    
}robj;

When the server wants to calculate the idle time of a key , The program will use the server's lrulock Attribute records the time minus the object's lru The time of the attribute record , The result is the idle time of the key

3.3, The update server executes commands per second

serverCron Function trackOperationsPerSecond The function takes every 100 Every millisecond , Estimate and record the number of command requests processed by the server in the last second by sampling calculation .

The reason why sampling calculation , Because the server will first calculate the average number of command requests processed per millisecond , And multiply the result by 1000, This gives an estimate of how many command requests the server can handle per second .

3.4, Update server memory peak record

Every time serverCron Function execution time , The program will check the amount of memory currently used by the server , And in stat_peak_memory Compare the saved values , If the amount of memory currently used is greater than stat_peak_memory The value of attribute record should be larger , Then the program will record the current amount of memory used to stat_peak_memory Attributes inside

struct redisServer{
    
    
    //...
    
    // Peak memory used 
	size_t stat_peak_memory;
    
    //...
    
};

3.5, Handle SIGTERM The signal

【 When the user Linux Server execution kill When the instruction is ready to kill a process , Will send SIGTERM The signal 】

When starting the server ,Redis Will be for the server process SIGTERM Signal correlation processor sigtermHandler function , This signal processor is responsible for receiving... From the server SIGTERM Signal time , Open the server state shutdown_asap identification ：

//SIGTERM The signal processor 
static void sigtermHandler(int sig){
    
    
    // Print log 
    redisLogFromHandler(REDIS_WARNING,"Received SIGTERM,scheduleing shutdown...");
    
    // Turn on and off the logo 
    server.shutdown_asap=1;
    
}

Every time serverCron Function runtime , The program will be on the server state shutdown_asap Property to check ：

• The value is 1, Shut down the server
• The value is 0, No action

serverCron You'll receive SIGTERM Signal time , To intercept , to RDB Persistence operation , Shut down the server . If one receives SIGTERM The signal turns off the server , Then the server cannot perform persistence operation .

3.6, Manage client resources

serverCron Every time the function executes, it will call clientsCron Function to manage client resources , This function does two things ：

• If the connection between the client and the server has timed out （ For a long time, there was no interaction between the client and the server ）, Then the program will release the client
• If the client is after the last command request , The size of the input buffer exceeds a certain length , Then the program will release the client's current input buffer , And recreate a default size input buffer , So as to prevent the input buffer of the client from consuming too much memory

3.7, Manage database resources

serverCron Function is called every time it is executed databasesCron function , This function checks part of the database in the server , Delete the expiration key , And when needed , Shrink the dictionary

3.8, Execution is delayed BGREWRITEAOF

Execute... On the server BGSAVE The duration of the order , If the client sends it to the server BGREWRITEAOF command , Then the server will BGREWRITEAOF The execution time of the command is delayed to BGSAVE After the command is executed .【 The reason why the execution is delayed BGREWRITEAOF It's because of performance considerations 】

Server's aof_rewrite_scheduled Identification records whether the server is delayed BGREWRITEAOF command , If the value is 1, Express BGREWRITEAOF The order was delayed

struct redisServer{
    
  
    //...
    
    // If the value is 1, So it means there is BGREWRITEAOF The command was delayed 
    int aof_rewrite_scheduled;
    
    //...
};

Each time the serverCron Function execution time , Will check BGSAVE Order or BGREWRITEAOF Whether the order is being executed , If it is not being implemented , also aof_rewrite_scheduled The value of the property is 1, Then the server will execute the delayed execution BGREWRITEAOF command

3.9, Check the running state of the persistence operation

Server state usage rdb_child_pid Properties and aof_child_pid Property record execution BGSAVE Command and BGREWRITEAOF The subprocess of the command ID, These two properties can also be used to check BGSAVE Order or BGREWRITEAOF Whether the order is being executed .

struct redisServer{
    
  
    //...
    
    // Record execution BGSAVE The subprocess of the command ID
    // If the server does not execute BGSAVE command , So the value of this property is -1
    pid_t rdb_child_pid;
    
    // Record execution BGREWRITEAOF The child process of the command id
    // If the server is not executing BGREWRITEAOF, So the value of this property is -1
    pid_t aof_child_pid;
    
    //...
};

Every time serverCron Function execution time , The program will check rdb_child_pid and aof_child_pid The value of two properties , As long as the value of one of the attributes is not -1（ Either execute on behalf of the server BGSAVE command , Or execute BGREWRITEAOF command ）, The program will be executed once wait3 function , Check whether the child process sends a signal to the server process ：

• If a signal arrives , So new RDB The file has been generated （ about BGSAVE Command, ） perhaps AOF The file has been rewritten （ about BGREWRITEAOF Command, ）, The server needs to perform subsequent operations of corresponding commands , Like with a new one RDB File to replace existing RDB file , Or use the rewritten AOF File to replace existing AOF file
• If no signal arrives , That means the persistence operation has not been completed , The program does not act

If the value of both attributes is -1, Then it means that the server is not performing persistence , In this case , The program will perform the following three checks ：

1. Check to see if there is BGREWRITEAOF Delayed , If there is , So start a new BGREWRITEAOF operation （ The inspection mentioned above ）
2. Check whether the automatic saving conditions of the server have been met , If the conditions are met , And the server is not performing other persistence operations , So the server starts a new BGSAVE operation （ Because of the conditions 1 May cause a BGREWRITEAOF, Therefore, in this check, it will confirm again whether the server is already in the process of persistence ）
3. Check the server settings AOF Whether the rewriting conditions meet , If the conditions are met , And the server is not performing other persistence operations , Then the server will start a new BGREWRITEAOF operation （ Because of the conditions 1 and 2 May cause new persistence operations , So in this examination , You need to confirm again whether the server is already performing persistence ）

Schematic diagram of the whole process ：

3.10, take AOF The contents of the buffer are written to AOF file

If the server is turned on AOF Persistence function , also AOF Data to be written in the buffer , that serverCron Function will call the corresponding program , take AOF The contents of the buffer are written to AOF In the document

3.11, Close asynchronous client

The server shuts down clients whose output buffer size exceeds the limit

3.12, increase cronloops The value of the counter

Server state cronloops Properties record serverCron The number of times the function executes ：

struct redisServer{
    
  
    //...
    
    //serverCron The counter for the number of runs of the function 
    //serverCron Every time a function is executed , The value of this attribute is increased by one 
    int cronloops;
    
    //...
};

4, Initialize server

Redis The server can accept the command request of the client from Startup , It needs to go through a series of initialization and setting processes ： For example, initialize the server state , Accept the server configuration specified by the user , Create corresponding data structure and network connection, etc . Detailed steps ：

1. So let's create one struct redisServer Instance variable of type server As the state of the server , And set default values for properties in the structure
   1. In this step, we will call initServerConfig function , Set up the running of the server ID、 Default operating frequency 、 Default profile path 、 Run the architecture 、 Default port number 、 Default RDB Persistence conditions and AOF Persistence conditions, etc ; And initialize the server LRU The clock ; Create a command table
   2. In this step initServerConfig The server status attributes set are basically integers 、 Floating point numbers 、 String attributes, etc . Besides the command list ,initServerConfig No data structures other than server state are created
2. Carry on initServerConfig, Load configuration options , The options in the specified configuration file are server Variable setting related attribute values
3. perform initServer, Create other data structures , And allocate memory for it
   1. For example, save the linked list of clients connected to the server ：server.clients; All databases in the server are saved ：server.db And so on are carried out in this step
   2. Server will server The initialization of the state is carried out in two steps ：initServerConfig Responsible for initializing general properties ;initServer Function is mainly responsible for initializing the data structure 【 The reason for this is , To avoid loading configuration options in the configuration file , Yes initServer The data structure created in . So load the configuration file first , Then initialize the data structure 】
   3. In addition to creating and initializing data structures ,initServer Also responsible for some very important setting operations . for example ： Create shared objects 、 Set up the process signal processor 、 Open the listening port of the server
4. After completing the server status server After initialization of variables , The server needs to load RDB Documents or AOF file , And restore the database state of the server according to the contents of the file records
   1. If the server is turned on AOF Persistence function , Then it will use AOF Restore the database state of files
   2. If the server is not turned on AOF Persistence function , Then it will use RDB Restore the database state of files
5. The last step of initialization , The server will print the log information after initialization , And start executing the server's event loop

The above steps can be summarized as ：

1. Initialize server state
2. Load server configuration
3. Initialize the server data structure
4. Restore database state
5. Execute the event loop