Redis source code and design analysis -- 15. RDB persistence mechanism

Redis RDB Persistence mechanism

1. RDB Introduce

because Redis Is a memory database , It stores its database state in memory , So if you don't find a way to save the database state stored in memory to disk , So once the server process exits , The state of the database in the server will also disappear .

To solve this problem ,Redis Provides RDB Persistence function , It will generate a compressed binary file , Through this file, you can restore the generated RDB The database state at the time of the file .

1.1 RDB Advantages and disadvantages

advantage

• RDB Is a compact binary , representative Redis Data snapshot at a point in time . Ideal for backup , Panoramic reproduction , Large scale data recovery and other scenarios .

• Redis load RDB Recover data much faster than AOF The way .

shortcoming

• BGSAVE Every time the command runs, it calls fork() Create child process , This is a heavyweight operation , It will occupy a certain amount of memory space .
• because RDB It is to perform persistence operations at regular intervals , If Redis Unexpected downtime , It may cause the last data not to be persisted in time .
• RDB Files are saved in a specific binary format ,Redis In the process of version evolution , There are many. RDB edition , This leads to version compatibility problems .

2. RDB Trigger mechanism

RDB It is divided into automatic trigger and manual trigger .

Manual trigger

There are two commands for manual triggering ：

• SAVE： Block the current Redis The server , until RDB Until the process is complete .
• BGSAVE：Redis Process execution fork() Operation creates a child process , In the background RDB The process of persistent operation （ Mainly use this ）.

Automatic triggering

The following three lines are configured in Redis In the configuration file ：

save 900 1  #  The server 900 In seconds , The database is at least 1 Time modification 
save 300 10 #  The server 300 In seconds , The database is at least 10 Time modification 
save 60 10000 #  The server 60 In seconds , The database is at least 10000 Time modification 

As long as any one of the above three conditions is satisfied ,BGSAVE The command will be executed .

3. RDB The implementation of the

About RDB Implementation in rdb.h Header files and rdb.c In the source file .

BGSAVE The implementation of the command is as follows ：

/* BGSAVE [SCHEDULE] */
// BGSAVE  Command implementation 
void bgsaveCommand(client *c) {
    
	// SCHEDULE control BGSAVE Implementation , Avoid and AOF Rewrite process conflict 
    int schedule = 0;

    if (c->argc > 1) {
    
        //  Set up schedule sign 
        if (c->argc == 2 && !strcasecmp(c->argv[1]->ptr,"schedule")) {
    
            schedule = 1;
        } else {
    
            addReply(c,shared.syntaxerr);
            return;
        }
    }

    //  If it is being implemented RDB Persistence operation , The exit 
    if (server.rdb_child_pid != -1) {
    
        addReplyError(c,"Background save already in progress");
    //  If it is being implemented AOF Persistence operation , Need to put BGSAVE Put on the agenda , And then quit 
    } else if (server.aof_child_pid != -1) {
    
        //  If schedule It's true , Set up rdb_bgsave_scheduled by 1
        if (schedule) {
    
            server.rdb_bgsave_scheduled = 1;
            addReplyStatus(c,"Background saving scheduled");
        } else {
    
            addReplyError(c,
                "An AOF log rewriting in progress: can't BGSAVE right now. "
                "Use BGSAVE SCHEDULE in order to schedule a BGSAVE whenver "
                "possible.");
        }

    //  perform BGSAVE
    } else if (rdbSaveBackground(server.rdb_filename) == C_OK) {
    
        addReplyStatus(c,"Background saving started");
    } else {
    
        addReply(c,shared.err);
    }
}

We can find out , When it can be executed BGSAVE During operation , The program calls a rdbSaveBackground function , This function will fork() A subprocess , Then the subprocess calls rdbSave() Conduct RDB Persistence , At the same time, the parent process will set some status information and update some log information .

//  Backstage RDB Persistence BGSAVE operation 
int rdbSaveBackground(char *filename) {
    
    pid_t childpid;
    long long start;

    //  There is no ongoing AOF and RDB operation , Otherwise return to C_ERR
    if (server.aof_child_pid != -1 || server.rdb_child_pid != -1) return C_ERR;

    //  Back up the dirty key values of the current database 
    server.dirty_before_bgsave = server.dirty;
    //  The last execution BGSAVE Time for 
    server.lastbgsave_try = time(NULL);
    // fork Function start time , Record fork Function takes time 
    start = ustime();
    //  Create child process 
    if ((childpid = fork()) == 0) {
    
        int retval;
        //  Code executed by child process 

        //  Close the listening socket 
        closeListeningSockets(0);
        //  Set process title , Easy to identify 
        redisSetProcTitle("redis-rdb-bgsave");
        //  Perform save operation , Write the database to filename In file 
        retval = rdbSave(filename);

        if (retval == C_OK) {
    
            //  Get the dirty private virtual page size of the child process , If you do RDB While the parent process is writing data , Then the child process will copy a copy of the memory of the parent process , Instead of sharing a share of memory with the parent process .
            size_t private_dirty = zmalloc_get_private_dirty();
            //  Write the content allocated by the child process to the log 
            if (private_dirty) {
    
                serverLog(LL_NOTICE,
                    "RDB: %zu MB of memory used by copy-on-write",
                    private_dirty/(1024*1024));
            }
        }
        //  Subprocess exit , Signal parent process , send out 0 Express BGSAVE success ,1 It means failure 
        exitFromChild((retval == C_OK) ? 0 : 1);
    } else {
    
        //  The code executed by the parent process 
        /* Parent */
        //  To calculate the fork Execution time of 
        server.stat_fork_time = ustime()-start;
        //  Calculation fork Rate ,GB/ Per second 
        server.stat_fork_rate = (double) zmalloc_used_memory() * 1000000 / server.stat_fork_time / (1024*1024*1024); /* GB per second. */
        // If fork The execution time , Exceeds the set threshold , Add it to a dictionary , And incoming "fork" relation , For delay diagnosis 
        latencyAddSampleIfNeeded("fork",server.stat_fork_time/1000);

        //  If fork error 
        if (childpid == -1) {
    
            server.lastbgsave_status = C_ERR;   // Set up BGSAVE error 
            //  Update log information 
            serverLog(LL_WARNING,"Can't save in background: fork: %s",
                strerror(errno));
            return C_ERR;
        }
        //  Update log information 
        serverLog(LL_NOTICE,"Background saving started by pid %d",childpid);
        server.rdb_save_time_start = time(NULL);    // Set up BGSAVE Start time 
        server.rdb_child_pid = childpid;            // Setup is responsible for execution BGSAVE Child process of operation id
        server.rdb_child_type = RDB_CHILD_TYPE_DISK;// Set up BGSAVE The type of , Write... To disk 
        // Turn off the hash table resize, because resize There will be copy actions in the process 
        updateDictResizePolicy();
        return C_OK;
    }
    return C_OK; /* unreached */
}

Subprocesses call rdbSave() The function will RDB File initialization , Just started to generate a temporary RDB file , Only after successful execution , It's going to happen rename operation , Then open the file with write permission , Then call rdbSaveRio() Function to write the contents of the database to temporary RDB file , Then flush the buffer and synchronize , Finally, close the file for rename Operate and update server status .

//  Save the database on disk , return C_OK success , Otherwise return to C_ERR
int rdbSave(char *filename) {
    
    char tmpfile[256];
    char cwd[MAXPATHLEN]; /* Current working dir path for error messages. */
    FILE *fp;
    rio rdb;
    int error = 0;

    //  Create temporary file 
    snprintf(tmpfile,256,"temp-%d.rdb", (int) getpid());
    //  Open the file in writing 
    fp = fopen(tmpfile,"w");
    //  Open the failure , Get file directory , Write to the log 
    if (!fp) {
    
        char *cwdp = getcwd(cwd,MAXPATHLEN);
        //  Write log information to logfile
        serverLog(LL_WARNING,
            "Failed opening the RDB file %s (in server root dir %s) "
            "for saving: %s",
            filename,
            cwdp ? cwdp : "unknown",
            strerror(errno));
        return C_ERR;
    }

    //  Initialize a rio object , This object is a file object IO
    rioInitWithFile(&rdb,fp);
    //  Write the contents of the database to rio in 
    if (rdbSaveRio(&rdb,&error) == C_ERR) {
    
        errno = error;
        goto werr;
    }

    /* Make sure data will not remain on the OS's output buffers */
    //  Flush buffer , Make sure all data is written to disk 
    if (fflush(fp) == EOF) goto werr;
    //  take fp The file pointed to is synchronized to disk 
    if (fsync(fileno(fp)) == -1) goto werr;
    //  Close file 
    if (fclose(fp) == EOF) goto werr;

    /* Use RENAME to make sure the DB file is changed atomically only * if the generate DB file is ok. */
    //  Atomicity changes rdb Name of file 
    if (rename(tmpfile,filename) == -1) {
    
        //  Failed to change the name , Then get the current directory path , Send log information , Delete temporary files 
        char *cwdp = getcwd(cwd,MAXPATHLEN);
        serverLog(LL_WARNING,
            "Error moving temp DB file %s on the final "
            "destination %s (in server root dir %s): %s",
            tmpfile,
            filename,
            cwdp ? cwdp : "unknown",
            strerror(errno));
        unlink(tmpfile);
        return C_ERR;
    }

    //  Write a log file 
    serverLog(LL_NOTICE,"DB saved on disk");
    //  Reset the dirty key of the server 
    server.dirty = 0;
    //  Update last SAVE Time of operation 
    server.lastsave = time(NULL);
    //  to update SAVE State of operation 
    server.lastbgsave_status = C_OK;
    return C_OK;

// rdbSaveRio() Function write error handling , Write the log , Close file , Delete temporary files , send out C_ERR
werr:
    serverLog(LL_WARNING,"Write error saving DB on disk: %s", strerror(errno));
    fclose(fp);
    unlink(tmpfile);
    return C_ERR;
}

Aforementioned rdbSave() Function opens the file , It will then call rdbSaveRio() function , This function will go to RDB Write fixed format content in the document ：

• REDIS Is a fixed string identifier , Occupy 5 byte .
• db_version yes RDB edition , Occupy 4 byte .
• Default message ： stay db_version And databases There is also default information , from rdbSaveInfoAuxFields() Function write , Include Redis edition 、Redis digit 、 Drink at the current time Redis Amount of memory currently in use .
• database Part contains zero or any database , And the key value pair data in each database .
• EOF Constant accounting 1 byte , End of file .
• check_sum It's the check sum , Occupy 8 byte .

rdbSaveRio() Function will traverse all databases , Then proceed RDB write in .

//  Will a RDB The contents of the format file are written to rio in , Successfully returns C_OK, otherwise C_ERR And some or all of the error messages 
//  When the function returns C_ERR, also error No NULL, that error Is set to an error code errno
int rdbSaveRio(rio *rdb, int *error) {
    
    dictIterator *di = NULL;
    dictEntry *de;
    char magic[10];
    int j;
    long long now = mstime();
    uint64_t cksum;

    //  The checksum option is enabled 
    if (server.rdb_checksum)
        //  Set the function of checksum 
        rdb->update_cksum = rioGenericUpdateChecksum;
    //  take Redis Save the version information to magic in 
    snprintf(magic,sizeof(magic),"REDIS%04d",RDB_VERSION);
    //  take magic writes rio in 
    if (rdbWriteRaw(rdb,magic,9) == -1) goto werr;
    //  take rdb The default information of the file is written to rio in 
    if (rdbSaveInfoAuxFields(rdb) == -1) goto werr;

    //  Traverse the databases in all servers 
    for (j = 0; j < server.dbnum; j++) {
    
    	//  Current database pointer 
        redisDb *db = server.db+j;
        //  When the key value of the database matches the dictionary 
        dict *d = db->dict;
        //  Skip empty database 
        if (dictSize(d) == 0) continue;
        //  Create an iterator of dictionary type 
        di = dictGetSafeIterator(d);
        if (!di) return C_ERR;

        /* Write the SELECT DB opcode */
        //  Write the selection identification code of the database  RDB_OPCODE_SELECTDB by 254
        if (rdbSaveType(rdb,RDB_OPCODE_SELECTDB) == -1) goto werr;
        //  Write to the database id, It takes up the length of one byte 
        if (rdbSaveLen(rdb,j) == -1) goto werr;

        //  Write the operation code of the adjustment database , We limit the size to UINT32_MAX within , This does not represent the actual size of the database , Just prompt to resize the hash table 
        uint32_t db_size, expires_size;
        //  If the size of the dictionary is larger than UINT32_MAX, Is set db_size For the biggest UINT32_MAX
        db_size = (dictSize(db->dict) <= UINT32_MAX) ?
                                dictSize(db->dict) :
                                UINT32_MAX;
        //  The size of the key with expiration time set exceeds UINT32_MAX, Is set expires_size For the biggest UINT32_MAX
        expires_size = (dictSize(db->expires) <= UINT32_MAX) ?
                                dictSize(db->expires) :
                                UINT32_MAX;
        //  Write the operation code for resizing the hash table ,RDB_OPCODE_RESIZEDB = 251
        if (rdbSaveType(rdb,RDB_OPCODE_RESIZEDB) == -1) goto werr;
        //  Write two values that prompt for resizing the hash table , If 
        if (rdbSaveLen(rdb,db_size) == -1) goto werr;
        if (rdbSaveLen(rdb,expires_size) == -1) goto werr;

        /* Iterate this DB writing every entry */
        //  Traverse all key value pairs in the database 
        while((de = dictNext(di)) != NULL) {
    
            sds keystr = dictGetKey(de);        // The current key 
            robj key, *o = dictGetVal(de);      // The value of the current key 
            long long expire;

            //  Create a key object in the stack and initialize 
            initStaticStringObject(key,keystr);
            //  Expiration time of current key 
            expire = getExpire(db,&key);
            //  The key object of the key , The value object , The expiration time is written to rio in 
            if (rdbSaveKeyValuePair(rdb,&key,o,expire,now) == -1) goto werr;
        }
        dictReleaseIterator(di);    // Release iterators 
    }
    di = NULL; 

    //  Write a EOF code ,RDB_OPCODE_EOF = 255
    if (rdbSaveType(rdb,RDB_OPCODE_EOF) == -1) goto werr;

    // CRC64 Inspection and , When the checksum is calculated as 0, Not open yes , Loading rdb File will be skipped 
    cksum = rdb->cksum;
    memrev64ifbe(&cksum);
    if (rioWrite(rdb,&cksum,8) == 0) goto werr;
    return C_OK;

//  Write error 
werr:
	//  Save error code 
    if (error) *error = errno;
    //  If the iterator is not released , Then release 
    if (di) dictReleaseIterator(di);
    return C_ERR;
}

The above said , The program will also call rdbSaveInfoAuxFields() Function to write some default auxiliary information ：

//  Will a rdb The default information of the file is written to rio in 
int rdbSaveInfoAuxFields(rio *rdb) {
    
    //  Judge the bus width of the host , yes 64 A still 32 position 
    int redis_bits = (sizeof(void*) == 8) ? 64 : 32;

    //  add to rdb Status information of the file ：Redis edition ,redis digit , Current time and Redis Amount of memory currently in use 
    if (rdbSaveAuxFieldStrStr(rdb,"redis-ver",REDIS_VERSION) == -1) return -1;
    if (rdbSaveAuxFieldStrInt(rdb,"redis-bits",redis_bits) == -1) return -1;
    if (rdbSaveAuxFieldStrInt(rdb,"ctime",time(NULL)) == -1) return -1;
    if (rdbSaveAuxFieldStrInt(rdb,"used-mem",zmalloc_used_memory()) == -1) return -1;
    return 1;
}

We can use od -cx dump.rdb Command to view saved in dump.rdb Contents of the file ：

Reference material ：

《Redis Design and implementation 》

Redis Source code analysis and comments