all the data of Redis is in memory. If it goes down suddenly, all the data will be lost. Therefore, persistence is needed to ensure that the data of Redis will not be lost due to failure. When Redis is restarted, the persistence file can be reloaded to recover the data;
Redis persistence related configuration
Configure in configuration file
###### aof ###### # redis.cnf appendonly no appendfilename "appendonly.aof" # aof policy selection: # appendfsync always appendfsync everysec # This is the default # appendfsync no # If Auto aof rewrite percentage is 0, aof replication is turned off auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb # yes if the aof data is incomplete, try to read the most data with correct format; # no if the aof data is incomplete and an error is reported, you can repair the aof file through redis check aof; aof-load-truncated yes # Enable hybrid persistence aof-use-rdb-preamble yes ###### rdb ###### # Close rdb save "" # save 3600 1 # save 300 100 # save 60 10000
Persistence mode and analysis
append only file
aof log stores the sequential instruction sequence of Redis server. aof log only records the instruction records of memory modification; Persistent storage is protocol data;
restore the state of the memory data structure of the current instance of Redis by replay ing the instruction sequence in the aof log
set key val # open aofappendonly yes # Turn off aof replication auto-aof-rewrite-percentage 0 # Turn off hybrid persistence aof-use-rdb-preamble no # Close rdbsave. rdb must be closed when aof is used save "" # Set aof file name appendfilename "appendonly.aof"
# 1. Each command swipes the disk, and the redis transaction is persistent # appendfsync always # 2. Brush the disk every second appendfsync everysec # 3. Leave it to the system to brush the disk # appendfsync no
as the time goes on, the aof logs will become longer and longer. If redis restarts, replaying the whole aof log will be very time-consuming, resulting in redis being unable to provide external services for a long time;
aof persistence policy will persist all modification commands; Many commands in it can be merged or deleted; For example:
# Meaningless in and out of the queue can be omitted lpush list a lpop list
aof rewrite on the basis of aof, if certain policies are met, the fork process will convert into a series of redis commands according to the current memory state and serialize them into a new aof log file. After serialization, the incremental aof logs occurred during the operation will be added to the new aof log file, and the old aof log file will be replaced after addition; In order to achieve the purpose of reducing aof logs;
the fork process has two modes: Command (bgrewriteaof) and configuration file.
Note: aof rewrite is enabled only if aof is enabled;
lpush list mark lpush list king lpush list darren bgrewriteaof # The above three commands will be combined into one command # Merge strategy: the number of elements contained in the key will be detected first. If there are more than 64, multiple commands will be used to record the value of the key; hset hash mark 10001 hset hash darren 10002 hset hash king 10003 hdel hash mark bgrewriteaof # At this time, mark will not appear in aof. Setting mark and deleting mark become like they have never been operated before
# Open aof appendonly yes # Enable aof replication auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb # Turn off hybrid persistence aof-use-rdb-preamble no # Close rdb save ""
# 1. redis will record the size of the last aof replication. If the cumulative size exceeds the original size, aof replication will occur; auto-aof-rewrite-percentage 100 # 2. In order to avoid multiple aof replication in strategy 1 when there is a small amount of data, strategy 2 needs more than 64mb before aof replication occurs on the premise of meeting strategy 1; auto-aof-rewrite-min-size 64mb
aof replication has achieved slimming based on aof, but the data volume of aof replication is still large; Loading will be very slow
rdb is a kind of snapshot Persistence Based on the disadvantage of large aof or aof replication files; Through the fork main process, it persistently stores the data key value pairs in the memory into the rdb file in the sub process; rdb stores compressed binary data;
# Turning off aof also turns off aof replication appendonly no # Turn off aof replication auto-aof-rewrite-percentage 0 # Turn off hybrid persistence aof-use-rdb-preamble no # Open rdb, that is, comment save "" # save "" # save 3600 1 # save 300 100 # save 60 10000
# The default redis policy is as follows: # Note: if multiple save policies are written, rdb persistence will be enabled if only one is satisfied # 1 modification in 3600 seconds save 3600 1 # 100 modifications in 300 seconds save 300 100 # 10000 modifications in 60 seconds save 60 10000
if rdb persistence is adopted, once redis goes down, redis will lose data for a period of time;
RDB often needs to fork the sub process to save the data set to the hard disk. When the data set is large, the fork process is very time-consuming, which may cause Redis to fail to respond to the client's request within some milliseconds. If the data set is huge and the CPU performance is not very good, this situation will last for 1 second. AOF also needs fork, but you can adjust the frequency of rewriting log files to improve the durability of the data set.
it is known from the above that rdb files are small and load quickly but lose more, while aof files are large and load slowly but lose less; Hybrid persistence is a persistence scheme that absorbs the advantages of rdb and aof; During aof replication, the actual persistent content is rdb. After persistence, the data modified during persistence is attached to the tail of the file in the form of aof;
hybrid persistence is actually optimized based on aof rewrite; Therefore, you need to start aof rewrite first;
# Open aof appendonly yes # Enable aof replication auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb # Enable hybrid persistence aof-use-rdb-preamble yes # Close rdb save "" # save 3600 1 # save 300 100 # save 60 10000
- In the MySQL caching scheme, redis does not enable persistence. Redis only stores hotspot data, which is based on MySQL; If you need to enable persistence when some data is accessed frequently, you can choose rdb persistence scheme at this time, which allows you to lose data for a period of time;
- It requires high data reliability. When the machine performance and memory are also safe (96g in the case of the worst copy during fork writing), redis can start aof and rdb at the same time. Note that it is not mixed persistence at this time; Redis restart gives priority to loading data from aof. Theoretically, aof contains more latest data; If only one is enabled, use mixed persistence;
- When loss is allowed, the primary redis can also be used for non persistence (96g, 90g) and the secondary redis can be used for persistence;
- Camouflage from the database;
Data security policy
Question: is it safe to copy persistent files?
it is secure. Once the persistent file is created, it will not be modified. When the server wants to create a new persistent file, it first saves the contents of the file in a temporary file. When the temporary file is written, the program uses rename(2) to replace the original persistent file with the temporary file atomically
Two issues should be considered for data security:
- Node downtime (redis is an in memory database, and downtime data will be lost)
- Disk failure
create a cron job to back up one RDB file to one folder every hour and one RDB file to another folder every day.
make sure that the backup of the snapshot has the corresponding date and time information. Each time you execute the periodic task script, use the find command to delete the expired snapshot: for example, you can keep the hourly snapshot in the last 48 hours and the daily snapshot in the last one or two months.
back up the RDB outside your data center at least once a day, or at least outside your physical machine running Redis server