background

This article mainly shares some problems found in the high concurrency of pressure measurement. The previous two articles have described some summarization and optimization of the message queue and database connection pool under the condition of high concurrency. Don't talk too much, get to the point.

Transactions, presumably, are not new to the king of CRUD. There are basically multiple requests that need to be written using transactions, while Spring's use of transactions is particularly simple. Only one @ Transactional annotation is needed, as shown in the following example:

    @Transactional
    public int createOrder(Order order){
        orderDbStorage.save(order);
        orderItemDbStorage.save(order.getItems());
        return order.getId();
    }

When we create an order, we usually need to put the order and the order item in the same transaction to ensure that they meet the ACID. Here, we just need to write a transaction annotation on the method of creating an order.

Fair use of transactions

For the above order creation code, if you need to add a new requirement now, send a message to the message queue or call an RPC after creating the order, what would you do? Many students will first think of calling directly in the transaction method:

    @Transactional
    public int createOrder(Order order){
        orderDbStorage.save(order);
        orderItemDbStorage.save(order.getItems());
        sendRpc();
        sendMessage();
        return order.getId();
    }

This kind of code will appear in many businesses written by many people. It's OK to nest rpc in transactions and some non DB operations. Once the non DB operations are slow or the traffic is large, large transactions will occur. Because the transaction is not committed all the time, the database connection will be occupied. At this time, you may ask, I just need to expand the number of database connections. If I can't expand the number of database connections, I will increase it to 1000. As mentioned in the previous article, the size of database connection pool will still affect the performance of our database. Therefore, database connection is not how much we want to expand.

How can we optimize it? We can think about it carefully here. Our non db operations do not satisfy our transaction ACID, so why write them in the transaction? So we can extract them here.

    public int createOrder(Order order){
        createOrderService.createOrder(order);
        sendRpc();
        sendMessage();
    }

In this method, first call the creation order of the transaction, and then call other non DB operations. If we want more complex logic now, for example, if the order is created successfully, we will send a successful RPC request and if it fails, we will send a failed RPC request. From the above code, we can do the following transformation:

    public int createOrder(Order order){
        try {
            createOrderService.createOrder(order);
            sendSuccessedRpc();
        }catch (Exception e){
            sendFailedRpc();
            throw e;
        }
    }

Usually we will catch exceptions, or carry out some special processing according to the return value. The implementation here needs to display the catch exceptions, and throw them at a time. This way is not very elegant, so how can we write this logic better?

TransactionSynchronizationManager

There are just a few tools and methods provided in Spring's transactions to help us fulfill this requirement. In the transaction synchronization manager, we provide the method to register the callBack for the transaction:

public static void registerSynchronization(TransactionSynchronization synchronization)
			throws IllegalStateException {

		Assert.notNull(synchronization, "TransactionSynchronization must not be null");
		if (!isSynchronizationActive()) {
			throw new IllegalStateException("Transaction synchronization is not active");
		}
		synchronizations.get().add(synchronization);
	}

Transaction synchronization, also known as the callBack of our transaction, provides us with some extension points:

public interface TransactionSynchronization extends Flushable {

	int STATUS_COMMITTED = 0;
	int STATUS_ROLLED_BACK = 1;
	int STATUS_UNKNOWN = 2;
	
	/**
	 * Triggered on suspend
	 */
	void suspend();

	/**
	 * Triggered when a pending transaction throws an exception
	 */
	void resume();


	@Override
	void flush();

	/**
	 * Trigger before transaction commit
	 */
	void beforeCommit(boolean readOnly);

	/**
	 * Triggered before the transaction completes
	 */
	void beforeCompletion();

	/**
	 * Triggered after transaction commit
	 */
	void afterCommit();

	/**
	 * Triggered after transaction completion
	 */
	void afterCompletion(int status);
}

We can use the aftercompletion method to implement our above business logic:

    @Transactional
    public int createOrder(Order order){
        orderDbStorage.save(order);
        orderItemDbStorage.save(order.getItems());
        TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronizationAdapter() {
            @Override
            public void afterCompletion(int status) {
                if (status == STATUS_COMMITTED){
                    sendSuccessedRpc();
                }else {
                    sendFailedRpc();
                }
            }
        });
        return order.getId();
    }

Here we directly implement afterCompletion, and judge which RPC we should send by the transaction status. Of course, we can further encapsulate TransactionSynchronizationManager.registerSynchronization to encapsulate it as a transaction Util, which can make our code more concise.

In this way, we don't have to write all non DB operations outside the method, so the code is more logical, more readable, and elegant.

The pit of afterCompletion

This callback code of registered transaction often appears in our business logic, such as refresh cache, send message queue, send notification message, etc. after a transaction is completed. In daily use, there is basically no problem with this code, but in the process of suppressing, we found that this block has a bottleneck, which takes a long time It was found that the waiting time for getting the connection from the database connection pool was long. Finally, we located the action of aftercompetition and did not return the database connection.

In Spring's AbstractPlatformTransactionManager, the code for commit processing is as follows:

private void processCommit(DefaultTransactionStatus status) throws TransactionException {
		try {
			boolean beforeCompletionInvoked = false;
			try {
				prepareForCommit(status);
				triggerBeforeCommit(status);
				triggerBeforeCompletion(status);
				beforeCompletionInvoked = true;
				boolean globalRollbackOnly = false;
				if (status.isNewTransaction() || isFailEarlyOnGlobalRollbackOnly()) {
					globalRollbackOnly = status.isGlobalRollbackOnly();
				}
				if (status.hasSavepoint()) {
					if (status.isDebug()) {
						logger.debug("Releasing transaction savepoint");
					}
					status.releaseHeldSavepoint();
				}
				else if (status.isNewTransaction()) {
					if (status.isDebug()) {
						logger.debug("Initiating transaction commit");
					}
					doCommit(status);
				}
				// Throw UnexpectedRollbackException if we have a global rollback-only
				// marker but still didn't get a corresponding exception from commit.
				if (globalRollbackOnly) {
					throw new UnexpectedRollbackException(
							"Transaction silently rolled back because it has been marked as rollback-only");
				}
			}
	

			// Trigger afterCommit callbacks, with an exception thrown there
			// propagated to callers but the transaction still considered as committed.
			try {
				triggerAfterCommit(status);
			}
			finally {
				triggerAfterCompletion(status, TransactionSynchronization.STATUS_COMMITTED);
			}

		}
		finally {
			cleanupAfterCompletion(status);
		}
	}

Here we only need to pay attention to the last few lines of code. We can find that our triggerAfterCompletion is the second last execution logic. When all the codes are executed, our cleanupAfterCompletion will be executed, and our return database connection is also in this code, which leads to our slow access to database connection.

How to optimize

How to optimize the above problems? There are three options for optimization:

• This method is the most primitive method above. It can extract some simple logic, but for some complex logic, such as the nesting of transactions, afterCompletion is called in the nesting, which will increase a lot of work and easily cause problems.
• It improves the speed of database connection pool return by multithreading asynchronously, which is suitable for writing at the end of transaction when registering afterCompletion, and directly putting what needs to be done in other threads. But if after completion appears between our transactions, such as nested transactions, it will lead to the subsequent business logic and transaction parallelism.
• Simulate Spring transaction callback registration to implement new annotations. The above two methods have their own disadvantages, so in the end, we adopt this method to implement a custom annotation @ MethodCallBack, which is printed on the top of the transaction, and then through similar registration code.

    @Transactional
    @MethodCallBack
    public int createOrder(Order order){
        orderDbStorage.save(order);
        orderItemDbStorage.save(order.getItems());
        MethodCallbackHelper.registerOnSuccess(() -> sendSuccessedRpc());
         MethodCallbackHelper.registerOnThrowable(throwable -> sendFailedRpc());
        return order.getId();
    }

Through the third method, we can use it normally only by replacing the places where we register transaction callbacks.

On big business

What is a big deal after all this talk? The simple point is that the transaction takes a long time to run, so it is a big transaction. Generally speaking, the factors that lead to long transaction time are as follows:

• There are many data operations, such as inserting a lot of data into a transaction, so the transaction execution time will naturally become very long.
• There is a big competition for locks. When all connections operate on the same data at the same time, there will be queue waiting, and the transaction time will naturally become longer.
• There are other non DB operations in the transaction, such as some RPC requests. Some people say that my RPC is very fast and will not increase the running time of the transaction, but RPC request itself is an unstable factor. Affected by many factors, network fluctuation and slow response of downstream services. If these factors occur, there will be a lot of long transaction time, which may cause Mysql to hang up And cause an avalanche.

In the above three cases, the first two may not be very common, but there are many non DB operations in the third transaction, which is very common for us. Usually, the reason for this situation is that we are used to norms. Beginners or some inexperienced people write code, often write a big method first, and add transaction annotation directly to this method , and then add to it, whatever logic it is, a shuttle, just like the following picture:

Of course, there are some people who want to do distributed transactions. Unfortunately, they use the wrong method. For distributed transactions, you can pay attention to Seata, and also use an annotation to help you do distributed transactions.

Last

In fact, at the end of the day, why is this problem? It is generally understood that it is done after completion. The database connection must have been released long ago, but this is not the case. Therefore, we can't live up to the letter when we use many API s. If there is no detailed doc, you should know more about the implementation details.

Of course, I hope that we should try our best not to shuttle before writing code, and take every code seriously.