Redis - handling of expired keys during RDB, AOF and replication

When you execute the SAVE command or BGSAVE command to create a new RDB file, the program will check the keys in the database, and the expired keys will not be saved to the newly created RDB file.
For example, if the database contains three keys k1, k2 and k3, and k2 has expired, when the SAVE command or BGSAVE command is executed, the program will only SAVE the data of k1 and k3 to the RDB file, and k2 will be ignored.
Therefore, the inclusion of expired keys in the database does not affect the generation of new RDB files.

rdb.c rdbsave() source code

/* Iterate this DB writing every entry 
         *
         * Traverse the database and write data for each key value pair
         */
        while((de = dictNext(di)) != NULL) {
            sds keystr = dictGetKey(de);
            robj key, *o = dictGetVal(de);
            long long expire;
            
            // According to keystr, create a key object in the stack
            initStaticStringObject(key,keystr);

            // Gets the expiration time of the key
            expire = getExpire(db,&key);

            // Save key value pair data
            if (rdbSaveKeyValuePair(&rdb,&key,o,expire,now) == -1) goto werr;
        }

The rdbSaveKeyValuePair function is implemented as follows

/* Save a key-value pair, with expire time, type, key, value.
 *
 * Writes the key, value, expiration time, and type of the key value pair to RDB.
 *
 * On error -1 is returned.
 *
 * Error returned - 1.
 *
 * On success if the key was actually saved 1 is returned, otherwise 0
 * is returned (the key was already expired). 
 *
 * 1 is returned when the key is saved successfully, and 0 is returned when the key has expired.
 */
int rdbSaveKeyValuePair(rio *rdb, robj *key, robj *val,
                        long long expiretime, long long now)
{
    /* Save the expire time 
     *
     * Expiration time of save key
     */
    if (expiretime != -1) {
        /* If this key is already expired skip it 
         *
         * Do not write expired keys
         */
        if (expiretime < now) return 0;

        if (rdbSaveType(rdb,REDIS_RDB_OPCODE_EXPIRETIME_MS) == -1) return -1;
        if (rdbSaveMillisecondTime(rdb,expiretime) == -1) return -1;
    }
    /* Save type, key, value 
     *
     * Save type, key, value
     */
    if (rdbSaveObjectType(rdb,val) == -1) return -1;
    if (rdbSaveStringObject(rdb,key) == -1) return -1;
    if (rdbSaveObject(rdb,val) == -1) return -1;
    return 1;
}

Load RDB file
When the Redis server is started, if the RDB function is enabled, the server will load the RDB file:

If the server runs in the master server mode, the program will check the keys saved in the file when loading the RDB file, the unexpired keys will be loaded into the database, and the expired keys will be ignored, so the expired keys will not affect the master server loading the RDB file;
If the server is running in slave server mode, when loading an RDB file, all keys saved in the file, whether expired or not, are loaded into the database. However, when the master-slave server synchronizes data, the database of the slave server will be cleared. Generally speaking, the expiration key will not affect the slave server loading RDB files;

This part of the code can be viewed in RDB Source code of rdbLoad() function in C:

		/* Check if the key already expired. This function is used when loading
         * an RDB file from disk, either at startup, or when an RDB was
         * received from the master. In the latter case, the master is
         * responsible for key expiry. If we would expire keys here, the
         * snapshot taken by the master may not be reflected on the slave. 
         *
         * If the server is the primary node,
         * When the keys have expired, they are no longer associated with the database
         */
        if (server.masterhost == NULL && expiretime != -1 && expiretime < now) {
            decrRefCount(key);
            decrRefCount(val);
            // skip
            continue;
        }

AOF file write
When the server runs in AOF persistence mode, if a key in the database has expired, but it has not been deleted lazily or periodically, the AOF file will not be affected by the expired key.

After the expired key is deleted lazily or periodically, the program will append a DEL command to the AOF file to explicitly record that the key has been deleted.
For example, if the client uses the GET message command to access the expired message key, the server will perform the following three actions:
1) Remove the message key from the database.
2) Append a DEL message command to the AOF file. (according to the added features of AOF files, AOF only has this DEL operation when the client makes a request)
3) Returns a null reply to the client executing the GET command.

This part is the use of the expireifneed function in the lazy deletion strategy in Redis. The inert deletion strategy is described in Redis inert deletion strategy. So I won't repeat it here.

It should be noted that the expireIfNeeded function is in dB The C / lookupKeyRead() function is called. The lookupKeyRead function is used to fetch the value of the key in the database db when reading.

AOF override
Similar to generating RDB files, during AOF rewriting, the program will check the keys in the database, and the expired keys will not be saved to the rewritten AOF file.

For example, if the database contains three keys k1, k2 and k3, and k2 has expired, the program will only rewrite k1 and k3, while k2 will be ignored.

If you master the method of AOF rewriting in this part, you will naturally understand it.

copy
When the server is running in replication mode, the deletion of the expiration key from the server is controlled by the master server:

  • After deleting an expiration key, the master server will explicitly send a DEL command to all slave servers to tell the slave server to delete the expiration key;
  • When the slave server executes the read command sent by the client, even if it encounters the expired key, it will not delete the expired key, but continue to process the expired key like the unexpired key;
  • The slave server will delete the expired key only after receiving the DEL command from the master server.

Keywords: Redis

Added by stylefrog on Wed, 12 Jan 2022 09:08:29 +0200