How to handle duplicate / concurrent requests gracefully?

In some cases, some user requests may be sent repeatedly. If it is a query operation, it does not matter, but some of them involve write operations. Once repeated, it may lead to serious consequences. For example, if the transaction interface is repeated, it may place orders repeatedly.

Repeated scenarios may be:

The hacker intercepted the request and replayed it
The front end / client has repeatedly sent the request for some reason, or the user has repeatedly clicked in a very short time.
Gateway retransmission
....

This paper discusses how to prevent users from clicking repeatedly and other client operations if this situation is handled gracefully and uniformly on the server.

De duplication with unique request number

You may think that as long as the request has a unique request number, you can borrow redis to do this de duplication - as long as the unique request number exists in redis and it is proved that it has been processed, it is considered to be repeated

The code is as follows:

    String KEY = "REQ12343456788";//Request unique number
    long expireTime =  1000;// After 1000 ms expires, repeated requests within 1000 ms will be considered repeated
    long expireAt = System.currentTimeMillis() + expireTime;
    String val = "expireAt@" + expireAt;

    //If the redis key still exists, the request is considered to be duplicate
    Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) connection -> connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));

    final boolean isConsiderDup;
    if (firstSet != null && firstSet) {// First visit
        isConsiderDup = false;
    } else {// redis value already exists. It is considered to be a duplicate
        isConsiderDup = true;
    }

De duplication of business parameters

The above scheme can solve the scenario with a unique request number. For example, before each write request, the server returns a unique number to the client. The client makes a request with this request number, and the server can complete de re interception.

However, in many scenarios, the request does not carry such a unique number! Can we use the request parameters as the identification of a request?

Consider a simple scenario first. Assuming that the request parameter has only one field reqParam, we can use the following identification to judge whether the request is repeated. User ID: interface name: request parameter

1	String KEY = "dedup:U="+userId + "M=" + method + "P=" + reqParam;

Then, when the same user accesses the same interface and comes with the same reqParam, we can locate that he is duplicate.

But the problem is that our interface is usually not so simple. In the current mainstream, our parameter is usually a JSON. So how can we redo this scene?

Calculate the summary of request parameters as parameter identification

Suppose we sort the request parameters (JSON) in ascending order according to the KEY, and then spell them into a string as the KEY value? But this may be very long, so we can consider finding a summary of MD5 as the parameter for this string to replace the position of reqParam.

1	String KEY = "dedup:U="+userId + "M=" + method + "P=" + reqParamMD5;

In this way, the unique identification of the request is marked!

Note: MD5 may be repeated theoretically, but de duplication is usually de duplication in a short time window (for example, one second). It is almost impossible for the same user to spell out different parameters for the same interface in a short time.

Continue to optimize and consider eliminating some time factors

The above problem is actually a good solution, but some problems may be found when it is actually put into use: some requesting users click repeatedly in a short time (for example, three requests are sent in 1000 milliseconds), but the above de duplication judgment (different KEY values) is bypassed.

The reason is that in the fields of these request parameters, there is a time field. This field marks the time of the user's request. The server can discard some old requests (for example, 5 seconds ago). As shown in the following example, other parameters of the request are the same, except that the request time is one second different:

    //The two requests are the same, but the request time is one second different
    String req = "{\n" +
            "\"requestTime\" :\"20190101120001\",\n" +
            "\"requestValue\" :\"1000\",\n" +
            "\"requestKey\" :\"key\"\n" +
            "}";

    String req2 = "{\n" +
            "\"requestTime\" :\"20190101120002\",\n" +
            "\"requestValue\" :\"1000\",\n" +
            "\"requestKey\" :\"key\"\n" +
            "}";

For this kind of request, we may also need to block the subsequent repeated requests. Therefore, such time fields need to be eliminated before business parameter summary. A similar field may be the longitude and latitude field of GPS (there may be a small difference between repeated requests).

Request de duplication tool class, Java implementation

public class ReqDedupHelper {

    /**
     *
     * @param reqJSON Request parameters, usually JSON
     * @param excludeKeys Which fields should be removed from the request parameters before the summary
     * @return MD5 summary of removal parameters
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decreptParam = reqJSON;

        TreeMap paramTreeMap = JSON.parseObject(decreptParam, TreeMap.class);
        if (excludeKeys!=null) {
            List<String> dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }

        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        String res = null;
        try {
            MessageDigest messageDigest = MessageDigest.getInstance("MD5");
            byte[] mdBytes = messageDigest.digest(src.getBytes());
            res = DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("",e);
        }
        return res;
    }
}

Here are some test logs:

public static void main(String[] args) {
    //The two requests are the same, but the request time is one second different
    String req = "{\n" +
            "\"requestTime\" :\"20190101120001\",\n" +
            "\"requestValue\" :\"1000\",\n" +
            "\"requestKey\" :\"key\"\n" +
            "}";

    String req2 = "{\n" +
            "\"requestTime\" :\"20190101120002\",\n" +
            "\"requestValue\" :\"1000\",\n" +
            "\"requestKey\" :\"key\"\n" +
            "}";

    //All parameters are compared, so the two parameters MD5 are different
    String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
    String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
    System.out.println("req1MD5 = "+ dedupMD5+" , req2MD5="+dedupMD52);

    //The comparison of removal time parameters is the same as MD5
    String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req,"requestTime");
    String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2,"requestTime");
    System.out.println("req1MD5 = "+ dedupMD53+" , req2MD5="+dedupMD54);

}

Log output:

1 2	req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267 , req2MD5=A2D20BAC78551C4CA09BEF97FE468A3F req1MD5 = C2A36FED15128E9E878583CAAAFEFDE9 , req2MD5=C2A36FED15128E9E878583CAAAFEFDE9

Log Description:

Since the requestTime of the two parameters is different at the beginning, it can be found that the two values are different when calculating the duplicate parameter summary
In the second call, the requestTime is removed and then the summary is obtained (the "requestTime" is passed in the second parameter). It is found that the two summaries are the same and meet the expectations.

summary

So far, we can get a complete de duplication solution, as follows:

String userId= "12345678";//user
String method = "pay";//Interface name
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req,"requestTime");//Calculate the request parameter summary, which eliminates the interference of the request time
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;

long expireTime =  1000;// After 1000 ms expires, repeated requests within 1000 ms will be considered repeated
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;

// NOTE: direct SETNX does not support with expiration time, so setting + expiration is not an atomic operation. In extreme cases, it may not expire if it is set. Later, the same request may be mistaken for de duplication. Therefore, the underlying API is used here to ensure that SETNX + expiration time is an atomic operation
Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) connection -> connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime),
        RedisStringCommands.SetOption.SET_IF_ABSENT));

final boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false;
} else {
    isConsiderDup = true;
}

Added by Broniukas on Tue, 04 Jan 2022 11:17:33 +0200

Programming VIP