Why persist

  all the data of Redis is in memory. If it goes down suddenly, all the data will be lost. Therefore, persistence is needed to ensure that the data of Redis will not be lost due to failure. When Redis is restarted, the persistence file can be reloaded to recover the data;

Redis persistence related configuration

Configure in configuration file

###### aof ######
# redis.cnf
appendonly no
appendfilename "appendonly.aof"

# aof policy selection:
# appendfsync always
appendfsync everysec # This is the default
# appendfsync no

# If Auto aof rewrite percentage is 0, aof replication is turned off
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
# yes if the aof data is incomplete, try to read the most data with correct format;
# no if the aof data is incomplete and an error is reported, you can repair the aof file through redis check aof;
aof-load-truncated yes
# Enable hybrid persistence
aof-use-rdb-preamble yes


###### rdb ######
# Close rdb
save ""

# save 3600 1
# save 300 100
# save 60 10000

Persistence mode and analysis

aof

append only file
  aof log stores the sequential instruction sequence of Redis server. aof log only records the instruction records of memory modification; Persistent storage is protocol data;

recovery

  restore the state of the memory data structure of the current instance of Redis by replay ing the instruction sequence in the aof log

to configure

set key val
# open 
aofappendonly yes
# Turn off aof replication
auto-aof-rewrite-percentage 0
# Turn off hybrid persistence
aof-use-rdb-preamble no
# Close rdbsave. rdb must be closed when aof is used
save ""
# Set aof file name
appendfilename "appendonly.aof"

strategy

# 1. Each command swipes the disk, and the redis transaction is persistent
# appendfsync always
# 2. Brush the disk every second
appendfsync everysec
# 3. Leave it to the system to brush the disk
# appendfsync no

shortcoming

  as the time goes on, the aof logs will become longer and longer. If redis restarts, replaying the whole aof log will be very time-consuming, resulting in redis being unable to provide external services for a long time;

aof replication

  aof persistence policy will persist all modification commands; Many commands in it can be merged or deleted; For example:

# Meaningless in and out of the queue can be omitted
lpush list a
lpop list

  aof rewrite on the basis of aof, if certain policies are met, the fork process will convert into a series of redis commands according to the current memory state and serialize them into a new aof log file. After serialization, the incremental aof logs occurred during the operation will be added to the new aof log file, and the old aof log file will be replaced after addition; In order to achieve the purpose of reducing aof logs;
  the fork process has two modes: Command (bgrewriteaof) and configuration file.
Note: aof rewrite is enabled only if aof is enabled;

Command mode:

lpush list mark
lpush list king
lpush list darren
bgrewriteaof
# The above three commands will be combined into one command
# Merge strategy: the number of elements contained in the key will be detected first. If there are more than 64, multiple commands will be used to record the value of the key;

hset hash mark 10001
hset hash darren 10002
hset hash king 10003
hdel hash mark
bgrewriteaof
# At this time, mark will not appear in aof. Setting mark and deleting mark become like they have never been operated before

Configuration mode:

# Open aof
appendonly yes
# Enable aof replication
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
# Turn off hybrid persistence
aof-use-rdb-preamble no
# Close rdb
save ""

strategy

# 1. redis will record the size of the last aof replication. If the cumulative size exceeds the original size, aof replication will occur;
auto-aof-rewrite-percentage 100
# 2. In order to avoid multiple aof replication in strategy 1 when there is a small amount of data, strategy 2 needs more than 64mb before aof replication occurs on the premise of meeting strategy 1;
auto-aof-rewrite-min-size 64mb

shortcoming

  aof replication has achieved slimming based on aof, but the data volume of aof replication is still large; Loading will be very slow

rdb

  rdb is a kind of snapshot Persistence Based on the disadvantage of large aof or aof replication files; Through the fork main process, it persistently stores the data key value pairs in the memory into the rdb file in the sub process; rdb stores compressed binary data;

to configure

# Turning off aof also turns off aof replication
appendonly no
# Turn off aof replication
auto-aof-rewrite-percentage 0
# Turn off hybrid persistence
aof-use-rdb-preamble no
# Open rdb, that is, comment save ""
# save ""
# save 3600 1
# save 300 100
# save 60 10000

strategy

# The default redis policy is as follows:
# Note: if multiple save policies are written, rdb persistence will be enabled if only one is satisfied
# 1 modification in 3600 seconds
save 3600 1
# 100 modifications in 300 seconds
save 300 100
# 10000 modifications in 60 seconds
save 60 10000

shortcoming

  if rdb persistence is adopted, once redis goes down, redis will lose data for a period of time;
  RDB often needs to fork the sub process to save the data set to the hard disk. When the data set is large, the fork process is very time-consuming, which may cause Redis to fail to respond to the client's request within some milliseconds. If the data set is huge and the CPU performance is not very good, this situation will last for 1 second. AOF also needs fork, but you can adjust the frequency of rewriting log files to improve the durability of the data set.

Mixed persistence

  it is known from the above that rdb files are small and load quickly but lose more, while aof files are large and load slowly but lose less; Hybrid persistence is a persistence scheme that absorbs the advantages of rdb and aof; During aof replication, the actual persistent content is rdb. After persistence, the data modified during persistence is attached to the tail of the file in the form of aof;
  hybrid persistence is actually optimized based on aof rewrite; Therefore, you need to start aof rewrite first;

to configure

# Open aof
appendonly yes
# Enable aof replication
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
# Enable hybrid persistence
aof-use-rdb-preamble yes
# Close rdb
save ""
# save 3600 1
# save 300 100
# save 60 10000

application

1. In the MySQL caching scheme, redis does not enable persistence. Redis only stores hotspot data, which is based on MySQL; If you need to enable persistence when some data is accessed frequently, you can choose rdb persistence scheme at this time, which allows you to lose data for a period of time;
2. It requires high data reliability. When the machine performance and memory are also safe (96g in the case of the worst copy during fork writing), redis can start aof and rdb at the same time. Note that it is not mixed persistence at this time; Redis restart gives priority to loading data from aof. Theoretically, aof contains more latest data; If only one is enabled, use mixed persistence;
3. When loss is allowed, the primary redis can also be used for non persistence (96g, 90g) and the secondary redis can be used for persistence;
4. Camouflage from the database;

Data security policy

Question: is it safe to copy persistent files?
  it is secure. Once the persistent file is created, it will not be modified. When the server wants to create a new persistent file, it first saves the contents of the file in a temporary file. When the temporary file is written, the program uses rename(2) to replace the original persistent file with the temporary file atomically

Two issues should be considered for data security:

1. Node downtime (redis is an in memory database, and downtime data will be lost)
2. Disk failure

  create a cron job to back up one RDB file to one folder every hour and one RDB file to another folder every day.
  make sure that the backup of the snapshot has the corresponding date and time information. Each time you execute the periodic task script, use the find command to delete the expired snapshot: for example, you can keep the hourly snapshot in the last 48 hours and the daily snapshot in the last one or two months.
  back up the RDB outside your data center at least once a day, or at least outside your physical machine running Redis server