The Transactions and Persistence of the redis Actual War

1. Transaction description

(1) What is a transaction

A transaction is a bundle of things that are bound together and executed sequentially until they are successful, otherwise they are restored to their previous state

Transactions must be subject to ACID principles, which are atomicity, consistency, isolation, and durability, respectively.

Atomicity: When these instructions are operated on, either they are all executed successfully or they are not executed at all.As long as one of the instructions fails, all instructions fail, data is rolled back, back to the data state before the instructions were executed

Consistency: Transaction execution transforms data from one state to another, but remains stable for the integrity of the entire data

Isolation: Any changes to data that occur during the execution of the transaction should only exist in the transaction and have no effect on the outside world. Changes to data made by the transaction will only be displayed after the transaction commits, and the data will not be available to other transactions until the transaction commits

Persistence: When a transaction completes correctly, it is permanent for changes to data

(2) Common errors caused by concurrent transactions

The first type of missing updates: when a transaction is undone, overwrite the update data submitted by other transactions

Dirty Read: One transaction reads uncommitted update data from another transaction

Magic reading is also called virtual reading: a transaction executes two queries, and the second result set contains data that was not found in the first query or that some rows have been deleted, causing two inconsistent results, except that another transaction inserts or deletes data between the two queries

Non-repeatable reading: A transaction reads the data of the same row twice and the result is in a different state. Just in the middle another transaction updates the data. The results are different and cannot be trusted.

The second type of missing updates: is a special case of non-repeatable reading.If both things read the same line, then both write and commit, the changes made by the first thing will be lost

2. redis transaction

The redis transaction is implemented through five commands: MULTI, WATCH, UNWAYCH, EXEC, DISCARD

MUTIL command: To start a transaction, the client can continue to send any number of commands to the server. These commands will not be executed immediately, but will be placed in a queue. When the EXEC command is called, all the commands in the queue will be executed and it will always return OK

WATCH command: Keys are monitored until the user executes the EXEC command. If other clients replace, update or delete any monitored keys first, the transaction will fail and return an error when the user compartment executes the EXEC command (after which the user can choose to retry or abandon the transaction)

UNWATCH command: reset the link after the WATCH command is executed and before the EXEC command is executed

EXEC command: execute transaction command

DISCARD command: Client can cancel WATCH command name and empty all queued commands

3. redis transaction example

The following will illustrate the transaction process with an example of a commodity transaction


(1) Put commodities on the market

a. Use hashes to manage all user information in the market, including user names, money owned by users


b. Use collections (inventory:) to manage all commodity information for each user, including the name of the commodity


inventory:1 reports for User:1

c. Use ordered collections (market:) to manage goods put on the market

Procedure: Check if the inventory:1 package contains ItemL commodities--->Add ItemL commodities from inventory:1 package to the trading market--->Delete ItemL commodities from inventory:1 package

def list_item(conn,userid,goodsname,price):
    #Use ordered collections (market:) to manage goods put on the market
    #The key of the item is userid:goodsid
    inventory = 'inventory:%s' %(userid)
    user = 'User:%s' %(userid)
    goodsitem = '%s:%s' %(userid,goodsname)
    end = time.time() + 5
    pipe = conn.pipeline()
    while time.time() < end:
   #Monitor changes to packages
            if not pipe.sismember(inventory,goodsname):  #Check if the user still holds the item that will be put on the market
                return None
            pipe.zadd('market:',{goodsitem:price}) #Put commodities on the market
            pipe.srem(inventory, goodsname) #Remove the item from the user package
            print('Goods in the mall:',conn.zrange('market:', 0, -1,withscores=True))
            print('Post-market Users{}Goods:{}'.format(userid, conn.smembers(inventory)))
            return True
        except redis.exceptions.WatchError as e:
    return False



(2) Transaction: User:2 Purchases ItemL commodities in the trading market

Idea: Use watch to monitor the market and the buyer's personal information, then get the amount of money the buyer has and the price of the commodity market, and check if the buyer has enough money to buy the commodity. If the buyer does not have enough money, the program cancels the transaction. If the buyer has enough money, the program first transfers the money paid by the buyer to the seller, and thenRemove the sold goods from the commodity market.

def purchase_item(conn, buyerid, goodsname, sellerid):
    buyer = 'User:{}'.format(buyerid) #Buyer key
    seller = 'User:{}'.format(sellerid) #Seller key
    buy_inventory = 'inventory:{}'.format(buyerid)   #Buyer parcel key
    sell_inverntory = 'inventory:{}'.format(sellerid)   #Seller Package key
    goodsitem = '{}:{}'.format(sellerid,goodsname)  #Commodity key in the market
    end = time.time() + 5
    pipe = conn.pipeline()
    while time.time() < end:
  'market:',buyer)  #Monitor the commodity market and buyer's personal information
            funds = int(pipe.hget(buyer,'funds'))  #Get the money the buyer has
            print('Money in the hands of the buyer:', funds)
            price = pipe.zscore('market:', goodsitem)  #Get the price of the item
            if funds < price:
                return None
            pipe.hincrby(buyer, 'funds', int(-price))  #Buyer's money is down
            pipe.hincrby(seller, 'funds', int(price))  #Seller's Money Increase
            pipe.zrem('market:', goodsitem)
            print('Commodities in the trading market:', conn.zrange('market:', 0, -1, withscores=True))
            print('The money the buyer has now:',conn.hget(buyer, 'funds'))
            print('Buyer package contents:', conn.smembers(buy_inventory))
            print('The money the seller has now:',conn.hget(seller, 'funds'))
            return True
        except redis.exceptions.WatchError as e:
    return False


if __name__ == '__main__':
    conn = redis.Redis()
    purchase_item(conn, 2, 'ItemL', 1)

4. redis persistence

Redis provides two different persistence methods to store data from memory to hard disk. One is RDB persistence, which is based on the persistence of a database record in redis memory dump to an RDB on disk, and the other is AOF(append only file) persistence, which is based on the replication of the written command executed to disk.

(1) RDB persistence configuration

# Save the DB on disk:
#  Set the frequency at which sedis mirrors the database.
#  900 At least one in seconds (15 minutes) key Value change (then database save)--Persistence).
#  300 At least 10 in seconds (5 minutes) key Value change (then database save)--Persistence).
#  60 At least 10,000 in seconds (1 minute) key Value change (then database save)--Persistence).
save 900 1
save 300 10
save 60 10000

stop-writes-on-bgsave-error yes
# Whether to compress when performing a mirrored backup.yes: compression, but some cpu consumption is required.No: no compression, more disk space is required.
rdbcompression yes
# One CRC64 The check is placed at the end of the file when stored or loaded rbd There will be a 10 when the file%About this performance degradation, you can turn off this configuration item in order to maximize performance.
rdbchecksum yes
# File name of snapshot
dbfilename dump.rdb
# Directory where snapshots are stored
dir ./

(2) Ways to create snapshots

A. Clients can create a snapshot by sending besave commands to redis. For platforms that support bgsave commands (basically all platforms support it except the windows platform), redis calls fork to create a child process, which writes the snapshot to disk, and the parent process continues to process command requests

b.Clients can also send save commands to redis to create a snapshot. The redis server that receives the Save command will not respond to any other commands until the snapshot is created. The Save command is not commonly used. We usually use it only when there is not enough memory to support bgsave, or even when waiting for the persistence operation to complete.

c. When redis receives a request to shut down the server through the shutdown command, or when it receives the standard term command, it executes a save command, blocks all clients, stops executing any commands sent by clients, and shuts down the server after the Save command has been executed

d. When a redis server links to another redis server and sends sync synchronization commands to each other to start a replication operation, if the master server is not currently performing the bgsave operation, or if the master server has not just completed the bgsave operation, the master server will perform the bgsave operation

(3) AOF persistence configuration

# Whether to turn on AOF, turn off by default (no)
appendonly yes
# Specify AOF file name
appendfilename appendonly.aof
# Redis supports three different brush modes:
# appendfsync always #Force a write to disk every time a write command is received, which is the most guaranteed, complete persistence, but also the slowest, and is generally not recommended.
appendfsync everysec #Forcing disk writes every second is a recommended way to compromise performance and persistence.
# appendfsync no     #Completely dependent on OS write, usually about 30 seconds, best performance but least guaranteed persistence, not recommended.

#During log rewriting, command appending is not performed, but rather placed in a buffer to avoid conflicts with command appending on DISK IO.
#Set to yes means no fsync for new writes during rewrite, temporarily in memory, write after rewrite is complete, default is no, yes is recommended
no-appendfsync-on-rewrite yes 
#The current AOF file size automatically starts a new log rewrite process when it is twice the size of the AOF file from the last log rewrite.
auto-aof-rewrite-percentage 100
#The current AOF file initiates a new log rewrite process minimum to avoid frequent rewrites due to smaller file sizes when Reids is just started.
auto-aof-rewrite-min-size 64mb

(4) AOF Rewrite Principle

AOF rewrite, like RDB snapshot creation, cleverly utilizes write-time replication:

a. Redis executes fork(), now has both parent and child processes

b. The subprocess begins writing the contents of the new AOF file to the temporary file

c. For all newly executed write commands, the parent process adds these changes to the end of the existing AOF file while accumulating them in a memory cache so that the existing AOF file is safe even if it stops halfway through the override

d. When the child process finishes rewriting, it sends a signal to the parent process, which, upon receiving the signal, appends all the data in the memory cache to the end of the new AOF file

Note: The main process is blocked when appending data from the memory cache to the end of the new AOF file and rename




Keywords: Redis snapshot Database Windows

Added by christh on Sat, 18 Apr 2020 01:04:59 +0300