catalogue
Application of string data type in redis
Redis
redis (Remote Dictionary Service) Remote Dictionary Service, memory database and key value database are widely used
Redis command view: http://redis.cn/commands.html
redis application scenario
Record the number of friends' circle likes, comments and hits (hash)
Record the list of circle of friends (sort), so as to quickly display the list of circle of friends
Record the title, abstract, author and cover of the article for list page display (hash)
Record the likes user ID list and comment ID list of the circle of friends for display and de duplication count (zset)
Cache hotspot data to reduce database pressure (hash)
If the circle of friends talk id is an integer id, redis can be used to allocate the circle of friends talk id (counter) (string)
Record the friend relationship (set) through the intersection, union and difference set operation of the set
In the game business, the record of each game is stored (list)
Install compile & & start
it clone https://gitee.com/mirrors/redis.git -b 6.2 cd redis make make test make install # It is installed in / usr/local/bin by default # Redis server is a server program # Redis cli is a client program mkdir redis-data # Put the redis folder into redis Copy conf to redis data # Modify redis conf # requirepass change password 123456 # daemonize yes cd redis-data redis-server redis.conf # Access redis server through redis cli redis-cli -h 127.0.0.1 -a 123456
redis storage structure
Five data structures of Redis
string
Character array. The string is a dynamic string raw. When the length of the string is less than 1M, the capacity will be doubled; Over 1M, only 1M more each time; The maximum length of the string is 512M; Note: redis string is a binary security string; Can store pictures, binary protocols and other binary data;
string basic command
# Set the value of key SET key val # Get the value of key GET key # Perform an atomic plus one operation INCR key # Performs the operation of adding an integer to an atom INCRBY key increment # Perform atomic minus one operation DECR key # Performs the atomic subtraction of an integer DECRBY key decrement # If the key does not exist, it is equivalent to the SET command. When the key exists, do nothing SETNX key value # Delete key val key value pair DEL key # Set or clear the bit value of the value (string) of the key at offset. SETBIT key offset value # Returns the bit value of the string corresponding to the key at offset GETBIT key offset # Count the number of bit s whose string is set to 1 BITCOUNT key
string storage structure
If the string length is less than or equal to 20 and can be converted into an integer, use int storage;
If the string length is less than or equal to 44, use embstr to store;
If the string length is greater than 44, raw storage is used;
Application of string data type in redis
1. Object storage
SET role:10001 '{["name"]:"mark",["sex"]:"male",["age"]:30}'
GET role:10001
When storing the object model with string, the object data that is not often modified is stored.
2 accumulator
# Add 1 to the total number of reading statistics incr reads # Add 100 in total incrby reads 100
3 distributed lock
# Lock setnx lock 1 # Release lock del lock # 1. Exclusive function 2 Definition of locking behavior 3 Release behavior definition
Spin lock: an unfair lock, which is not obtained in the order of the requested lock
Mutex lock: a fair lock acquires locks in the order in which they are requested
redis implements non fair locks, while etcd zk implements fair locks
4-bit operation
# Monthly check-in function 10001 user ID 202106 check-in in June 2021 the first day of June setbit sign:10001:202106 1 1 # Calculate the attendance in June 2021 bitcount sign:10001:202106 # Get the check-in status on the second day of June 2021. 1 has checked in and 0 has not checked in getbit sign:10001:202106 2
String can generate bitmap. value:string can be used as a bitmap
setbit uses embsrt raw, so why not use int for string? Because string is a dynamic string, it can better save memory.
string has three encoding methods
If the string length is less than or equal to 20 and can be converted into an integer, use int storage; If the string length is less than or equal to 44, use embstr to store; If the string length is greater than 44, raw storage is used
list
Two way linked list implementation, list head and tail operation (delete and increase) time complexity O(1); The basis for finding whether the data in the list with O(n) time complexity of intermediate elements is compressed:
1. If the element length is less than 48, it will not be compressed; 2. The compression length of the front and back elements shall not exceed 8;
list basic command
# Queue one or more elements from the left side of the queue LPUSH key value [value ...] # Pop up an element from the left side of the queue LPOP key # Queue one or more elements from the right side of the queue RPUSH key value [value ...] # Pop up an element from the right side of the queue RPOP key # Returns the elements 0, 1, 2 from the queue between start and end LRANGE key start end # Remove the element with the value of the previous count from the list stored in the key storage structure application Stack (first in, last out) FILO) LREM key count value # It is a blocking version of RPOP, because this command blocks the connection when a given list cannot pop up any elements BRPOP key timeout # Timeout + delay queue
list storage structure
/* Minimum ziplist size in bytes for attempting compression. */
#define MIN_COMPRESS_BYTES 48
/* quicklistNode is a 32 byte struct describing a ziplist for a quicklist.
* We use bit fields keep the quicklistNode at 32 bytes.
* count: 16 bits, max 65536 (max zl bytes is 65k, so max count actually <
32k).
* encoding: 2 bits, RAW=1, LZF=2.
* container: 2 bits, NONE=1, ZIPLIST=2.
* recompress: 1 bit, bool, true if node is temporary decompressed for
usage.
* attempted_compress: 1 bit, boolean, used for verifying during testing.
* extra: 10 bits, free for future use; pads out the remainder of 32 bits */
typedef struct quicklistNode {
struct quicklistNode *prev;
struct quicklistNode *next;
unsigned char *zl;
unsigned int sz; /* ziplist size in bytes */
unsigned int count : 16; /* count of items in ziplist */
unsigned int encoding : 2; /* RAW==1 or LZF==2 */
unsigned int container : 2; /* NONE==1 or ZIPLIST==2 */
unsigned int recompress : 1; /* was this node previous compressed? */
unsigned int attempted_compress : 1; /* node can't compress; too small
*/
unsigned int extra : 10; /* more bits to steal for future usage */
} quicklistNode;
typedef struct quicklist {
quicklistNode *head;
quicklistNode *tail;
unsigned long count; /* total count of all entries in all
ziplists */
unsigned long len; /* number of quicklistNodes */
int fill : QL_FILL_BITS; /* fill factor for individual
nodes */
unsigned int compress : QL_COMP_BITS; /* depth of end nodes not to
compress;0=off */
unsigned int bookmark_count: QL_BM_BITS;
quicklistBookmark bookmarks[];
} quicklist;
list application
Stack (first in and last out FILO)
LPUSH + LPOP # perhaps RPUSH + RPOP
Queue (FIFO)
LPUSH + RPOP # perhaps RPUSH + LPOP
Blocking queue
LPUSH + BRPOP # perhaps RPUSH + BLPOP
Asynchronous message queue
The operation is the same as queue, but between different systems.
Get fixed window record (record)
# Cut the last 5 records and nearly 50 records ltrim says 0 4 lrange says 0 -1
The atomicity of commands needs to be guaranteed in actual projects, so lua script or pipeline command is generally used;
ltrim say 0 49 ltrim is a clipping operation
The lua virtual machine is nested inside redis
-- redis lua script
local record = KEYS[1]
redis.call("LPUSH", "says", record)
redis.call("LTRIM", "says", 0, 4)
hsah
Hash table contains this data structure c + + unorderd in many high-level languages_ Map quickly indexes value through key;
Basic command
# Get the value corresponding to the field in the hash corresponding to the key HGET key field # Set the value corresponding to the field in the hash corresponding to the key HSET key field value # Set multiple hash key value pairs HMSET key field1 value1 field2 value2 ... fieldn valuen # Get the value of multiple field s HMGET key field1 field2 ... fieldn # Add an integer value to the value corresponding to the field in the hash corresponding to the key HINCRBY key field increment # Get the number of key value pairs in the hash corresponding to the key HLEN key # Delete the key value pair of the hash corresponding to the key. The key is field HDEL key field
storage structure
If the number of nodes is greater than 512 or the length of all strings is greater than 64, use dict;
If the number of nodes is less than 512 and a string is less than 64, use ziplost (compressed list) to implement.
application
hmset hash:10001 name mark age 18 sex male
# Compare with string
set hash:10001 '{["name"]:"mark",["sex"]:"male",["age"]:18}'
# Suppose the age of mark is 19
# hash:
hset hash:10001 age 19
# string:
get role:10001
# Call json to decrypt the obtained string, take out the field and modify the age value
# Then call json encryption
set role:10001 '{["name"]:"mark",["sex"]:"male",["age"]:19}'hash stores the attributes of the object, and you can modify the attributes of the object at any time. However, if weak countries use string to store the attributes of the object, modifying the attributes of the object will be more troublesome and inefficient.
# Use user id as key # Commodity id as field # Quantity of goods as value # Note: these items are displayed in the order we add them; # Add item: hset MyCart:10001 40001 1 lpush MyItem:10001 40001 # Increase quantity: hincrby MyCart:10001 40001 1 hincrby MyCart:10001 40001 -1 // Reduce quantity 1 # Show all item quantities: hlen MyCart:10001 # Delete item: hdel MyCart:10001 40001 lrem MyItem:10001 1 40001 # Get all items: lrange MyItem:10001 # 40001 40002 40003 hget MyCart:10001 40001 hget MyCart:10001 40002 hget MyCart:10001 40003
set
Set: used to store unique fields. It is not required to be in order; The storage does not need to be orderly, and the operation needs to be orderly when intersecting and inserting sets.
Basic command
# Add one or more specified member elements to the key of the collection SADD key member [member ...] # Calculate the number of collection elements SCARD key # SMEMBERS key SMEMBERS key # Returns whether the member member is a member of the stored set key SISMEMBER key member # Randomly return one or more elements in the key set without deleting them SRANDMEMBER key [count] # Removes and returns one or more random elements from the collection stored in the key SPOP key [count] # Returns the element of the difference between a set and a given set SDIFF key [key ...] # Returns the intersection of members of all specified sets SINTER key [key ...] # Returns all members of a given set of multiple sets SUNION key [key ...]
Storage structure
If all elements are integers and the number of nodes is less than or equal to 512, the integer array is used for storage; If one of the elements is not an integer or the number of nodes is greater than 512, the dictionary is used for storage.
application
luck draw
# Add lottery user sadd Award:1 10001 10002 10003 10004 10005 10006 sadd Award:1 10009 # View all lottery users smembers Award:1 # Extract multiple lucky users srandmember Award:1 10 # If one first prize, two second prizes and three third prizes are selected, how to operate?
Common concern
sadd follow:A mark king darren mole vico sadd follow:C mark king darren sinter follow:A follow:C
Recommend friends
sadd follow:A mark king darren mole vico sadd follow:C mark king darren # C possible acquaintances: sdiff follow:A follow:C
zset
Ordered set; To realize the ranking again, it is an orderly and unique data structure
member maintains uniqueness and socre maintains order
Basic command
# Added to the sorted set with key ZADD key [NX|XX] [CH] [INCR] score member [score member ...] # Delete the key value pair of member from the ordered set with key as key ZREM key member [member ...] # Returns the score value of the member in the ordered set key ZSCORE key member # Add increment to the score value of the member of the ordered set key ZINCRBY key increment member # Returns the number of ordered set elements of the key ZCARD key # Returns the ranking of member s in the ordered set key ZRANK key member # Returns the element order by ID limit 1100 of the specified range stored in the ordered set key ZRANGE key start stop [WITHSCORES] # Returns the members in the specified interval in the ordered set key (in reverse order) ZREVRANGE key start stop [WITHSCORES]
storage structure
If the number of nodes is greater than 128 or the length of a string is greater than 64, use the skip list;
If the number of nodes is less than 128 and the length of all strings is less than 64, ziplost storage is used
Save space when there is little data o(n)
When the number is large, access performance O(1),O(logn)
application
Baidu hot list (Baidu ranking list)
Delay queue
The message queue is formed into a string as the member of zset, and the expiration processing time of the message is regarded as the socre. Then, multiple threads poll zset to obtain the expired tasks for processing.
def delay(msg):
msg.id = str(uuid.uuid4()) #Ensure member uniqueness
value = json.dumps(msg)
retry_ts = time.time() + 5 # Try again in 5s
redis.zadd("delay-queue", retry_ts, value)
# Use connection pool
def loop():
while True:
values = redis.zrangebyscore("delay-queue", 0, time.time(), start=0,
num=1)
if not values:
time.sleep(1)
continue
value = values[0]
success = redis.zrem("delay-queue", value)
if success:
msg = json.loads(value)
handle_msg(msg)
# Disadvantages: loop is a multi-threaded competition. Both threads get data from zrangebyscore, but zrem succeeds and loses
Defeat,
# Optimization: to avoid redundant operations, you can use lua script atom to execute these two commands
# Solution: funnel current limiting
Distributed timer
The producer hash es the scheduled tasks to different redis entities, and assigns a dispatcher process to each redis entity to obtain the timeout time in redis regularly and send it to different consumers.
Time window current limiting
The system limits that a user's behavior can only occur dynamically N times within the specified effort range
# Specify user_ A behavior action of ID can only occur for the number of times during a specific time period
max_count
local function is_action_allowed(red, userid, action, period, max_count)
local key = tab_concat({"hist", userid, action}, ":")
local now = zv.time()
red:init_pipeline()
-- Record behavior
red:zadd(key, now, now)
-- Remove the behavior records before the time window, and the rest are records in the time window
red:zremrangebyscore(key, 0, now - period *100)
-- Gets the number of behaviors in the time window
red:zcard(key)
-- Set the expiration time to avoid the length of the memory window occupied by cold users+1 second
red:expire(key, period + 1)
local res = red:commit_pipeline()
return res[3] <= max_count
# Clear the maintenance window once, and keep all the records in the window;
# Disadvantages: the data in all time windows are recorded. If this amount is large, it is not suitable for such current limiting; Funnel current limiting
# Note: if you use key + expire, it can also be realized, but the realization is fusing, and the maintenance time window is the function of current limiting