In daily development, there are many similar operations to reduce inventory, such as goods inventory in e-commerce system, prize inventory in lottery system, etc.
1 Solution
- Using the mysql database, use a field to store the inventory, each time the inventory is deducted to update the field.
- Use the database, but store multiple copies of the inventory in multiple records, and route them when deducting the inventory, which increases the concurrency, but you still cannot avoid a lot of access to the database to update the inventory.
- Put inventory in redis Use the incrby feature of redis to deduct inventory.
2 Analysis
The first and second ways above are to deduct inventory based on data.
2.1 Based on database inventory
The first way is to wait here for the lock on all requests to get the lock to deduct the inventory. It can be used with low concurrency, but once the concurrency is large, a large number of requests will be blocked here, causing the requests to time out and the entire system to avalanche. It also frequently accesses the database, which consumes a lot of database resources, so this method is not applicable when concurrency is high.
2.2 Database-based Multi-Inventory
The second way is actually the optimized version of the first way, which improves the concurrency to some extent, but still takes up a large amount of database resources to update the database.
2.2. 1 There are still some problems to reduce inventory based on database:
With database deduction of inventory, the operation of deducting inventory must be executed in a statement, and selec cannot be update d first. This will lead to overdeductions under concurrent circumstances. For example:
update number set x=x-1 where x > 0
MySQL has its own problems with high concurrent processing performance. Generally speaking, the processing performance of MySQL increases with concurrent threads, but after a certain degree of concurrency, there will be a significant inflection point, then all the way down, and eventually even worse than that of a single threads.
When inventory reduction and high concurrency occur together, the problem of contending for InnoDB row locks will occur because the number of operations in the same row, leading to waiting for each other or even deadlock, which greatly reduces the processing performance of MySQL and eventually results in a timeout exception on the front-end pages.
2.3 based on redis
For these problems, we have a third solution: put the inventory in the cache, use the incrby feature of redis to deduct the inventory, and solve the problem of overshoot and performance. However, once the cache is lost, a recovery scenario needs to be considered. For example, when the lottery system deducts the prize inventory, the initial inventory = the total inventory - the number of awards issued, but if the prize is awarded asynchronously, you need to wait until the MQ message is consumed before restarting redis to initialize the inventory, otherwise there is a problem of inventory inconsistency.
3 Realization of inventory deduction based on redis
We use redis'lua script to reduce inventory
Since it is a distributed environment, a distributed lock is required to control that only one service can initialize the inventory
A callback function needs to be provided to get the initial inventory when it is initialized
3.1 Initialize the Inventory Callback Function (IStockCallback)
/**
* Get Inventory Callback
* @author yuhao.wang
*/
public interface IStockCallback {
/**
* Get inventory
* @return
*/
int getStock();
}
3.2 StockService
/**
* Deduct inventory
*
* @author yuhao.wang
*/
@Service
public class StockService {
Logger logger = LoggerFactory.getLogger(StockService.class);
/**
* Unlimited inventory
*/
public static final long UNINITIALIZED_STOCK = -3L;
/**
* Redis Client
*/
@Autowired
private RedisTemplate<String, Object> redisTemplate;
/**
* Execute script to deduct inventory
*/
public static final String STOCK_LUA;
static {
/**
*
* @desc Deduction inventory Lua script
* stock-1: Indicates unlimited inventory
* Stock 0: means there is no stock
* stock greater than 0: indicates remaining inventory
*
* @params Inventory key
* @return
* -3:Inventory not initialized
* -2:Insufficient inventory
* -1:Unlimited inventory
* Greater than or equal to 0: Remaining inventory (after deduction)
* redis Cached inventory (value) is -1 for unlimited inventory, returning 1 directly
*/
StringBuilder sb = new StringBuilder();
sb.append("if (redis.call('exists', KEYS[1]) == 1) then");
sb.append(" local stock = tonumber(redis.call('get', KEYS[1]));");
sb.append(" local num = tonumber(ARGV[1]);");
sb.append(" if (stock == -1) then");
sb.append(" return -1;");
sb.append(" end;");
sb.append(" if (stock >= num) then");
sb.append(" return redis.call('incrby', KEYS[1], 0 - num);");
sb.append(" end;");
sb.append(" return -2;");
sb.append("end;");
sb.append("return -3;");
STOCK_LUA = sb.toString();
}
/**
* @param key Inventory key
* @param expire Inventory valid time, in seconds
* @param num Quantity deducted
* @param stockCallback Initialize Inventory Callback Function
* @return -2:Insufficient inventory; - 1: unlimited inventory; Greater than or equal to 0:Remaining inventory after deduction of inventory
*/
public long stock(String key, long expire, int num, IStockCallback stockCallback) {
long stock = stock(key, num);
// Initialize Inventory
if (stock == UNINITIALIZED_STOCK) {
RedisLock redisLock = new RedisLock(redisTemplate, key);
try {
// Acquire locks
if (redisLock.tryLock()) {
// Dual validation to avoid duplicate backsourcing to the database during concurrency
stock = stock(key, num);
if (stock == UNINITIALIZED_STOCK) {
// Get Initial Inventory
final int initStock = stockCallback.getStock();
// Set inventory to redis
redisTemplate.opsForValue().set(key, initStock, expire, TimeUnit.SECONDS);
// One-time deduction of inventory
stock = stock(key, num);
}
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
} finally {
redisLock.unlock();
}
}
return stock;
}
/**
* Add inventory (restore inventory)
*
* @param key Inventory key
* @param num Inventory quantity
* @return
*/
public long addStock(String key, int num) {
return addStock(key, null, num);
}
/**
* Add inventory
*
* @param key Inventory key
* @param expire Expiration time (seconds)
* @param num Inventory quantity
* @return
*/
public long addStock(String key, Long expire, int num) {
boolean hasKey = redisTemplate.hasKey(key);
// Determine if the key exists, update it directly if it exists
if (hasKey) {
return redisTemplate.opsForValue().increment(key, num);
}
Assert.notNull(expire,"Initialization of inventory failed, inventory expiration time cannot be null");
RedisLock redisLock = new RedisLock(redisTemplate, key);
try {
if (redisLock.tryLock()) {
// Once the lock is acquired, check again if there is a key
hasKey = redisTemplate.hasKey(key);
if (!hasKey) {
// Initialize Inventory
redisTemplate.opsForValue().set(key, num, expire, TimeUnit.SECONDS);
}
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
} finally {
redisLock.unlock();
}
return num;
}
/**
* Get inventory
*
* @param key Inventory key
* @return -1:Unlimited inventory; Greater than or equal to 0: Remaining inventory
*/
public int getStock(String key) {
Integer stock = (Integer) redisTemplate.opsForValue().get(key);
return stock == null ? -1 : stock;
}
/**
* Deduct inventory
*
* @param key Inventory key
* @param num Deduct inventory quantity
* @return Remaining inventory after deduction [-3: inventory not initialized; -2: insufficient inventory; -1: unlimited inventory; greater than or equal to 0: remaining inventory after deduction of inventory]
*/
private Long stock(String key, int num) {
// KEYS parameters in scripts
List<String> keys = new ArrayList<>();
keys.add(key);
// ARGV parameters in scripts
List<String> args = new ArrayList<>();
args.add(Integer.toString(num));
long result = redisTemplate.execute(new RedisCallback<Long>() {
@Override
public Long doInRedis(RedisConnection connection) throws DataAccessException {
Object nativeConnection = connection.getNativeConnection();
// Cluster mode and single machine mode execute scripts the same way, but they do not have a common interface and can only be executed separately
// Cluster mode
if (nativeConnection instanceof JedisCluster) {
return (Long) ((JedisCluster) nativeConnection).eval(STOCK_LUA, keys, args);
}
// standalone mode
else if (nativeConnection instanceof Jedis) {
return (Long) ((Jedis) nativeConnection).eval(STOCK_LUA, keys, args);
}
return UNINITIALIZED_STOCK;
}
});
return result;
}
}
3.3 Calls
/**
* @author yuhao.wang
*/
@RestController
public class StockController {
@Autowired
private StockService stockService;
@RequestMapping(value = "stock", produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
public Object stock() {
// Commodity ID
long commodityId = 1;
// Inventory ID
String redisKey = "redis_key:stock:" + commodityId;
long stock = stockService.stock(redisKey, 60 * 60, 2, () -> initStock(commodityId));
return stock >= 0;
}
/**
* Get initial inventory
*
* @return
*/
private int initStock(long commodityId) {
// TODO Here are some operations to initialize the inventory
return 1000;
}
@RequestMapping(value = "getStock", produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
public Object getStock() {
// Commodity ID
long commodityId = 1;
// Inventory ID
String redisKey = "redis_key:stock:" + commodityId;
return stockService.getStock(redisKey);
}
@RequestMapping(value = "addStock", produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
public Object addStock() {
// Commodity ID
long commodityId = 2;
// Inventory ID
String redisKey = "redis_key:stock:" + commodityId;
return stockService.addStock(redisKey, 2);
}
}