redis_day02 review
Five data types and application scenarios
| type | characteristic | Usage scenario |
|---|---|---|
| string | Simple key value type. Value can be string and number | Regular counting (microblog number, fan number and other functions) |
| hash | hash is a mapping table of string type field and value. hash is especially suitable for storing objects | Store some data that may need to be changed (such as user information) |
| list | Ordered repeatable list | Message queue, etc |
| set | Unordered non repeatable list | Store and calculate relationships (e.g. microblog, followers or fans are stored in the collection, and functions such as common concern and common preference can be realized through intersection, union, difference and other operations) |
| sorted set | A collection with a score for each element | Various leaderboards |
redis_day03 notes
affair
characteristic
1. Separate isolation operation: all commands in the transaction will be serialized and executed in order, and will not be interrupted by commands sent by other clients during execution 2. Atomicity is not guaranteed: redis If there is a command execution failure in a transaction, other commands will still be executed without rollback mechanism
Transaction command
1,MULTI # Start transaction MySQL begin 2,Command 1 # Execute command 3,Command 2 ... ... 4,EXEC # Submit to the database and execute MySQL commit 4,DISCARD # Cancel transaction MySQL 'rollback'
Use steps
# Open transaction 127.0.0.1:6379> MULTI OK # Command 1 queued 127.0.0.1:6379> INCR n1 QUEUED # Command 2 queued 127.0.0.1:6379> INCR n2 QUEUED # Commit to database for execution 127.0.0.1:6379> EXEC 1) (integer) 1 2) (integer) 1
Command error handling in transaction
# 1. The syntax of the command is wrong. The command fails to join the queue. The transaction will be automatically discard ed directly This command will cause an error before executing the call. For example, this command may have syntax errors (wrong number of parameters, wrong command name) Processing scheme: if there is a syntax error, it will be executed automatically discard Case: 127.0.0.1:6379[7]> MULTI OK 127.0.0.1:6379[7]> get a QUEUED 127.0.0.1:6379[7]> getsss a (error) ERR unknown command 'getsss' 127.0.0.1:6379[7]> 127.0.0.1:6379[7]> 127.0.0.1:6379[7]> EXEC (error) EXECABORT Transaction discarded because of previous errors. # 2. If the command syntax is correct, but the type operation is incorrect, the transaction fails after the call and cannot be rolled back We performed an error due to value of key Operation (e.g String Type value Implemented List Command operation) Processing scheme: occurs in EXEC Then there is no special way to deal with it: even if some commands fail in the transaction, other commands will be executed. case 127.0.0.1:6379> MULTI OK 127.0.0.1:6379> set num 10 QUEUED 127.0.0.1:6379> LPOP num QUEUED 127.0.0.1:6379> exec 1) OK 2) (error) WRONGTYPE Operation against a key holding the wrong kind of value 127.0.0.1:6379> get num "10" 127.0.0.1:6379>
Think about why redis does not support rollback?
Pipeline pipeline
Definition: execute redis commands in batches to reduce communication io
Note: This is client technology
Examples
import redis
# Create a connection pool and connect to redis
pool = redis.ConnectionPool(host = '127.0.0.1',db=0,port=6379)
r = redis.Redis(connection_pool=pool)
pipe = r.pipeline()
pipe.set('fans',50)
pipe.incr('fans')
pipe.incrby('fans',100)
pipe.execute()
Performance comparison
# Create a connection pool and connect to redis
pool = redis.ConnectionPool(host = '127.0.0.1',db=0,port=6379)
r = redis.Redis(connection_pool=pool)
def withpipeline(r):
p = r.pipeline()
for i in range(1000):
key = 'test1' + str(i)
value = i+1
p.set(key, value)
p.execute()
def withoutpipeline(r):
for i in range(1000):
key = 'test2' + str(i)
value = i+1
r.set(key, value)
python operation redis transaction
with r.pipeline(transaction=true) as pipe
pipe.multi()
pipe.incr("books")
pipe.incr("books")
values = pipe.execute()
watch - optimistic lock
Function: during a transaction, you can listen to the specified key. When the command is submitted, if the value corresponding to the monitored key is not modified, the transaction can be submitted successfully, otherwise it will fail
> watch books OK > multi OK > incr books QUEUED > exec # Transaction execution failed (nil) watch After that, open another terminal to enter redis > incr books # Modify book value (integer) 1
python operation watch
#Operate on one account at the same time, current balance * 2
Data persistence
Persistence definition
Place data from power down volatile memory on permanent storage devices
Why persistence
Because all the data is in memory, it must be persistent
RDB mode (on by default)
1,Save real data 2,Save all database data contained in the server to the hard disk in the form of binary files 3,Default file name:/var/lib/redis/dump.rdb
Two ways to create rdb files
**Method 1: * * use SAVE or BGSAVE command in redis terminal
127.0.0.1:6379> SAVE OK # characteristic 1,implement SAVE During the command, redis The server will be blocked and cannot process the command request sent by the client SAVE After the command is executed, the server will restart processing the command request sent by the client 2,If RDB If the file already exists, the server will automatically use the new file RDB The file replaces the old one RDB file # A file is persisted regularly during work 127.0.0.1:6379> BGSAVE Background saving started # The execution process is as follows 1,Client send BGSAVE To server 2,The server will return immediately Background saving started To client 3,The server fork() Subprocesses do this 4,The server continues to provide services 5,Child process created RDB Inform after filing Redis The server # Profile related /etc/redis/redis.conf 263 that 's ok: dir /var/lib/redis # Indicates the rdb file storage path 253 that 's ok: dbfilename dump.rdb # file name # Comparison of two commands SAVE than BGSAVE Fast, because you need to create child processes and consume additional memory # Add: you can view the log files to see what redis has done # Log file: search logfile in configuration file logfile /var/log/redis/redis-server.log
Method 2: automatically save the configuration file when the conditions are met (most used)
# redis profile default
218 that 's ok: save 900 1
219 that 's ok: save 300 10
Indicates if the distance from the last creation RDB If the file has passed 300 seconds and all databases of the server have been modified no less than 10 times in total, it will be executed automatically BGSAVE command
220 that 's ok: save 60 10000
1,As long as any one of the three conditions is met, the server will execute automatically BGSAVE
2,Every time you create RDB After the file is saved, the time counter and times counter set by the server for automatic persistence will be cleared and the counting will be restarted, so the effects of multiple save conditions will not be superimposed
# This configuration item can also be executed on the command line [not recommended]
redis>save 60 10000
RDB disadvantages
1,establish RDB The file needs to save all the database data of the server, which is a very resource and time-consuming operation, so the server needs to create a new database after a period of time RDB File, that is, create RDB Files cannot be executed too frequently, otherwise it will seriously affect the performance of the server 2,Possible data loss
AOF(AppendOnlyFile)
1,Commands are stored, not real data 2,Not on by default # Opening method (modify configuration file) 1,/etc/redis/redis.conf 672 that 's ok: appendonly yes # Change no to yes 676 that 's ok: appendfilename "appendonly.aof" 2,Restart service sudo /etc/init.d/redis-server restart
AOF persistence principle and advantages
# principle 1,Whenever a command to modify the database is executed, 2,because AOF All database modification commands executed by the server are stored in the file, so one is given AOF File, the server just needs to execute it again AOF All commands contained in the file can achieve the purpose of restoring the database # advantage Users can make adjustments according to their own needs AOF Persistence is adjusted so that Redis In the event of an unexpected outage, no data is lost, or only one second of data is lost, which is better than RDB Persistence loses much less data
Special instructions
# because Although the server executes a command to modify the database, it will write the executed command to AOF File, but that doesn't mean AOF File persistence will not lose any data. In the current common operating system, execute system calls write Function, when writing some content to a file, in order to improve efficiency, the system usually does not write the content directly to the hard disk, but puts the content into a memory cache( buffer)The contents stored in the buffer are not really written to the hard disk until the buffer is filled # therefore 1,AOF Persistence: when a command is actually written to the hard disk, the command will not be accidentally lost due to shutdown 2,AOF The number of commands that persistence loses when it encounters downtime depends on the time when the commands are written to the hard disk 3,The earlier commands are written to the hard disk, the less data is lost in the event of an unexpected shutdown, and vice versa
Policy - Profile
# Open the configuration file: / etc / redis / redis Conf, find the relevant policies as follows 1,701 that 's ok: alwarys Every time the server writes a command, it writes the command in the buffer to the hard disk. Even if the server stops unexpectedly, it will not lose any command data that has been successfully executed 2,702 that 's ok: everysec(# (default) The server writes the commands in the buffer to the hard disk every second. In this mode, even if the server encounters an unexpected shutdown, only 1 second of data will be lost at most 3,703 that 's ok: no The server does not actively write commands to the hard disk,The operating system decides when to write the commands in the buffer to the hard disk. The number of lost commands is uncertain # Running speed comparison always: Slow speed everysec and no Very fast, the default value is everysec
AOF override
Think: will there be many redundant commands in the AOF file?
To make AOF The file size shall be controlled within a reasonable range to avoid random growth, redis Provided AOF Rewrite function, through which the server can generate a new AOF file -- new AOF The database data recorded in the file and the reason AOF The database data recorded in the file is exactly the same -- new AOF The file uses as few commands as possible to record database data, so it is new AOF References to documents are usually much smaller -- AOF During rewriting, the server will not be blocked and can normally process command requests sent by the client
Examples
| Original AOF file | Rewritten AOF file |
|---|---|
| select 0 | SELECT 0 |
| sadd myset peiqi | SADD myset peiqi qiaozhi danni lingyang |
| sadd myset qiaozhi | SET msg 'hello tarena' |
| sadd myset danni | RPUSH mylist 2 3 5 |
| sadd myset lingyang | |
| INCR number | |
| INCR number | |
| DEL number | |
| SET message 'hello world' | |
| SET message 'hello tarena' | |
| RPUSH mylist 1 2 3 | |
| RPUSH mylist 5 | |
| LPOP mylist |
AOF override - trigger
1,Client sends to server BGREWRITEAOF command
127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started
2,Modify the configuration file for the server to execute automatically BGREWRITEAOF command
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
# explain
1,Only when AOF The increment of the file is greater than 100%It is only rewritten when it is, that is, it is triggered when it is twice as large
# New for the first rewrite: 64M
# The second rewrite added: 128M
# Third rewrite: 256M (128M added)
Comparison of RDB and AOF persistence
| RDB persistence | AOF persistence |
|---|---|
| Full backup, saving the whole database at one time | Incremental backup, save one command to modify the database at a time |
| Long save interval | The save interval defaults to one second |
| Fast data restore | The data restore speed is average, there are many redundant commands, and the restore speed is slow |
| When the SAVE command is executed, the server will be blocked, but BGSAVE triggered manually or automatically will not block the server | No matter at ordinary times or during AOF rewriting, the server will not be blocked |
# Redis is used to store real data. Each one can't be lost. always is used. Some do caching and some save real data. I can open multiple redis services. Different businesses use different persistence. Sina has four redis services on each server. There are thousands of redis services in the whole business. Each persistence level is different for different businesses.
Data recovery (no manual operation)
Existing dump.rdb，Again appendonly.aof，Who to call when recovering? Look first appendonly.aof
Profile common configuration summary
# Set password 1,requirepass password # Open remote connection 2,bind 127.0.0.1 ::1 Comment out 3,protected-mode no Put the default yes Change to no # rdb persistence - default configuration 4,dbfilename 'dump.rdb' 5,dir /var/lib/redis # rdb persistence - Auto trigger (conditional) 6,save 900 1 7,save 300 10 8,save 60 10000 # aof persistence on 9,appendonly yes 10,appendfilename 'appendonly.aof' # aof persistence strategy 11,appendfsync always 12,appendfsync everysec # default 13,appendfsync no # aof override trigger 14,auto-aof-rewrite-percentage 100 15,auto-aof-rewrite-min-size 64mb # Set to slave server 16,salveof <master-ip> <master-port>
Storage path of Redis related files
1,configuration file: /etc/redis/redis.conf 2,Backup file: /var/lib/redis/*.rdb|*.aof 3,log file: /var/log/redis/redis-server.log 4,Startup file: /etc/init.d/redis-server # /Store the configuration file under etc / # /etc/init.d / store the service startup file
Redis master-slave replication
- definition
1,One Redis A service can have multiple copies of the service Redis Service becomes master，Other copies become slaves 2,master Will always synchronize their own data updates to slaves，Keep master-slave synchronization 3,only master You can execute the write command, slave Only read commands can be executed
- effect
Shared the pressure of reading (high concurrency)
- principle
Execute the read command sent by the client from the server, such as GET,LRANGE,SMEMMBERS,HGET,ZRANGE Wait, the client can connect slaves Execute read requests to reduce master Read pressure
- Implementation mode
-
Mode 1 (Linux command line implementation)
redis-server --slaveof --masterauth
# Slave server redis-server --port 6300 --slaveof 127.0.0.1 6379 # From client redis-cli -p 6300 127.0.0.1:6300> keys * # It is found that the data in redis of the original 6379 port has been copied 127.0.0.1:6300> set mykey 123 (error) READONLY You can't write against a read only slave. 127.0.0.1:6300> # Only data can be read from the server, not written
-
Mode 2 (Redis command line implementation)
# Two commands 1,>slaveof IP PORT 2,>slaveof no one # Server start redis-server --port 6301 # Client connection tarena@tedu:~$ redis-cli -p 6301 127.0.0.1:6301> keys * 1) "myset" 2) "mylist" 127.0.0.1:6301> set mykey 123 OK # Switch to from 127.0.0.1:6301> slaveof 127.0.0.1 6379 OK 127.0.0.1:6301> set newkey 456 (error) READONLY You can't write against a read only slave. 127.0.0.1:6301> keys * 1) "myset" 2) "mylist" # Then switch to main 127.0.0.1:6301> slaveof no one OK 127.0.0.1:6301> set name hello OK
-
Mode 3 (using configuration file)
# Each redis service has a corresponding configuration file # Two redis services 1,6379 -> /etc/redis/redis.conf 2,6300 -> /home/tarena/redis_6300.conf # Modify profile vi redis_6300.conf slaveof 127.0.0.1 6379 port 6300 # Start redis service redis-server redis_6300.conf # Client connection test redis-cli -p 6300 127.0.0.1:6300> hset user:1 username guods (error) READONLY You can't write against a read only slave.
-
Question: what if the master hangs up?
1,One Master There can be more than one Slaves 2,Slave Offline, but the processing performance of read requests decreases 3,Master Offline, write request cannot be executed 4,One of them Slave use SLAVEOF no one Command become Master，other Slaves implement SLAVEOF The command points to this new Master，Synchronize data from it # The above process is manual and automatic, which requires Sentinel sentry to realize Failover
demonstration
1,Boot port 6400 redis，Set to 6379 slave redis-server --port 6400 redis-cli -p 6400 redis>slaveof 127.0.0.1 6379 2,Boot port 6401 redis，Set to 6379 slave redis-server --port 6401 redis-cli -p 6401 redis>slaveof 127.0.0.1 6379 3,Close 6379 redis sudo /etc/init.d/redis-server stop 4,Put 6400 redis Set to master redis-cli -p 6400 redis>slaveof no one 5,Put 6401 redis Set to 6400 redis of salve redis-cli -p 6401 redis>slaveof 127.0.0.1 6400 # This is a manual operation, which is inefficient and takes time. Is there an automatic???
Sentinel sentry
sentinel of Redis - sentinel
1,Sentinel Will keep checking Master and Slaves Is it normal 2,every last Sentinel Any number of can be monitored Master And this Master Lower Slaves
Case demonstration
* * 1. Environment construction
# There are three redis services in total
1,Start 6379 redis The server
sudo /etc/init.d/redis-server start
2,Start 6380 redis Server, set to 6379 slave
redis-server --port 6380
tarena@tedu:~$ redis-cli -p 6380
127.0.0.1:6380> slaveof 127.0.0.1 6379
OK
3,Start 6381 redis Server, set to 6379 slave
redis-server --port 6381
tarena@tedu:~$ redis-cli -p 6381
127.0.0.1:6381> slaveof 127.0.0.1 6379
* * 2, * * installation and erection of sentinel sentry
# 1. Install redis Sentinel sudo apt install redis-sentinel verification: sudo /etc/init.d/redis-sentinel stop # 2. Create a new configuration file sentinel conf port 26379 sentinel monitor tedu 127.0.0.1 6379 1 # 3. Start sentinel Mode 1: redis-sentinel sentinel.conf Mode II: redis-server sentinel.conf --sentinel #4. Terminate the redis service of the master to see if it will be promoted to the master sudo /etc/init.d/redis-server stop # It is found that 6381 is upgraded to master and the other two are slave # Set a new value on 6381 and view it on 6380 127.0.0.1:6381> set name tedu OK # Start 6379 and observe the log. It is found that the slave is changed to 6381 Master-slave+Sentinels are basically enough
sentinel.conf interpretation
# sentinel listening port. The default is 26379. It can be modified port 26379 # Tell sentinel to listen to a master whose address is ip:port. The master name here can be customized. quorum is a number indicating how many sentinel think a master is invalid. The master is really invalid sentinel monitor <master-name> <ip> <redis-port> <quorum> #If the master has a password, you need to add this configuration sentinel auth-pass <master-name> <password> #How long does the master lose contact before it is considered unavailable? The default is 30 seconds sentinel down-after-milliseconds <master-name> <milliseconds>
python get master
from redis.sentinel import Sentinel
#Generate sentinel connection
sentinel = Sentinel([('localhost', 26379)], socket_timeout=0.1)
#Initialize master connection
master = sentinel.master_for('tedu', socket_timeout=0.1, db=1)
slave = sentinel.slave_for('tedu',socket_timeout=0.1, db=1)
#Use redis related commands
master.set('mymaster', 'yes')
print(slave.get('mymaster'))