In the project, Redis should not be used as a traditional database; Store a large amount of data without expiration time. If a large number of keys without expiration time and invalid keys are stored; In addition, the expiration policy of Redis itself is not set correctly, which will occupy a lot of memory. This will result in insufficient memory resources. Recently, I encountered such a situation in the project.

This is generally the case. Redis secondary storage (a self-made concept) is adopted in the project. The general meaning is to use UUID to generate a UUID as value first; This UUID is used as a version id; The key stored in the cache is ACL_CACHE_VERSION_KEY, i.e

SET ACL_CACHE_VERSION_KEY uuid

Then, the uuid is not used as part of another key. ACL_USER_ + uuid, for example, if the value of uuid is 26a26b84-578d-40bf-ab15-aeb188a56393, then the key is ACL_USER_26a26b84-578d-40bf-ab15-aeb188a56393, the data cached to the key is

HMSET ACL_USER_26a26b84-578d-40bf-ab15-aeb1 id1 12345 id2 45678

Because ACL_ USER_ Whether the 26a26b84-578d-40bf-ab15-aeb188a56393 is expired is after a new version of uuid is generated in the program and new user authority data is stored; Delete in program. Because the program is not robust, a large number of expired versions are not deleted in time. After years of accumulation, there are a large number of invalid versions of key s in Redis.

In the test environment, you can use the keys command to vaguely query the required keys, but this operation is only suitable for use in the test environment and not in the production environment. The reason is that Redis runs in a single thread. When the amount of data in Redis is large, this operation will traverse all data and return all the results at one time, resulting in a long execution time, As a result, subsequent operations are waiting, which directly affects the normal operation of the system. The solution is to use the scan command:

scan cursor [MATCH pattern] [COUNT count]

• Cursor: indicates a cursor, starting from 0. A new cursor value will be returned after this command is executed. If cursor= "0" means that there are still keys that have not been returned. You need to call scan again and use this new cursor value to obtain the next batch of keys; If cursor=="0", the traversal ends.
• pattern: a style that represents a fuzzy match
• count: indicates the maximum number of records returned in a batch. The default value is 10

Code implementation using jedis in cluster environment

import java.util.HashSet;
import java.util.Set;

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;
import redis.clients.jedis.JedisPoolConfig;
import redis.clients.jedis.ScanParams;
import redis.clients.jedis.ScanResult;

public class RedisTester {

    public static JedisCluster getJedisCluster() {
        JedisPoolConfig config = new JedisPoolConfig();
        config = new JedisPoolConfig();
        config.setMaxTotal(60000);// Set maximum connections
        config.setMaxIdle(1000); // Set maximum idle number
        config.setMaxWaitMillis(3000);// Set timeout
        config.setTestOnBorrow(true);

        // Cluster node
        Set<HostAndPort> jedisClusterNode = new HashSet<HostAndPort>();
        jedisClusterNode.add(new HostAndPort("yun-1.cacheyum.com", 6379));
        jedisClusterNode.add(new HostAndPort("yun-2.cacheyum.com", 6379));
        jedisClusterNode.add(new HostAndPort("yun-3.cacheyum.com", 6379));
        jedisClusterNode.add(new HostAndPort("yun-4.cacheyum.com", 6379));
        jedisClusterNode.add(new HostAndPort("yun-5.cacheyum.com", 6379));
        jedisClusterNode.add(new HostAndPort("yun-6.cacheyum.com", 6379));
        JedisCluster jc = new JedisCluster(jedisClusterNode, 1000, 1000, 1000, "admin321", config);
        return jc;
    }

    public static void scanCluster() {
        JedisCluster redis = getJedisCluster();
        redis.getClusterNodes().values().stream().forEach(pool -> {
            boolean done = false;
            String cur = "0";
            try (Jedis jedisNode = pool.getResource()) {
                while (!done) {
                    ScanParams scanParams = new ScanParams();
                    scanParams.match(" ACL_USER_*");
                    scanParams.count(10);
                    ScanResult<String> resp = jedisNode.scan(cur, scanParams);
                    for (String result : resp.getResult()) {
                        System.out.println("key: " + result);
                    }
                    cur = resp.getStringCursor();
                    System.out.println("cursor: " + cur);
                    if (cur.equals("0")) {
                        done = true;
                    }
                }
            }
        });
    }

    public static void main(String[] args) {
        scanCluster();
    }

}

Implementation of jedis code in single machine

import java.util.HashSet;
import java.util.Set;

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPoolConfig;
import redis.clients.jedis.ScanParams;
import redis.clients.jedis.ScanResult;

public class RedisTester {


    public static Jedis getAloneRedis() {
        Jedis jedis = new Jedis("127.0.0.1", 6379);
        System.out.println("connect successfully");
        jedis.auth("admin321");
        return jedis;
    }

    public void scanAlone() {
        Jedis jedis = getAloneRedis();
        String cursor = "0";
        do {
            ScanParams scanParams = new ScanParams();
            scanParams.match("ACL_USER_*");
            scanParams.count(10);
            ScanResult<String> sr = jedis.scan(cursor, scanParams);
            List<String> resultList = sr.getResult();
            for (String result : resultList) {
                System.out.println("key: " + result);
            }
            cursor = sr.getStringCursor();
            System.out.println("cursor: " + cursor);
        } while (!cursor.equals("0"));
    }

    public static void main(String[] args) {
        getAloneRedis();
    }

}

Use the following code to find out which versions of uuid are invalid. After finding it, call del instruction to delete it; Or to be more secure, call expire and add an expiration time. You can also disable the key within a certain time.