scene

After a machine crashes in the redis cluster, the redis of the background service will always report an error and cannot connect to the redis cluster. View the redis cluster through the command. It is found that the redis cluster cluster is normal. The standby slave machine has been upgraded to master.

Conjecture and verification

It is preliminarily speculated that the spring redis connection pool framework did not refresh the connection of the connection pool after one of the redis master machines crashed, but still connected to the hung redis server By looking for information, change Spring boot2.x to use Lettuce framework and redis by default The connection found the relevant information about RedisCluster in the Lettuce official document <Refreshing the cluster topologyview> Adaptive topology refresh Updates) and periodic updates are turned off by default and can be turned on by code.

Verification

• The official code example is as follows:

    RedisClusterClient clusterClient =
    RedisClusterClient.create(RedisURI.create("localhost", 6379));
  
    ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                                    .enablePeriodicRefresh(10, TimeUnit.MINUTES)
                                    .build();
  
    clusterClient.setOptions(ClusterClientOptions.builder()
                                    .topologyRefreshOptions(topologyRefreshOptions)
                                    .build());
    ...
  
    clusterClient.shutdown();
  
  
    RedisURI node1 = RedisURI.create("node1", 6379);
    RedisURI node2 = RedisURI.create("node2", 6379);
  
    RedisClusterClient clusterClient = RedisClusterClient.create(Arrays.asList(node1, node2));
  
    ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                                    .enableAdaptiveRefreshTrigger(RefreshTrigger.MOVED_REDIRECT, RefreshTrigger.PERSISTENT_RECONNECTS)
                                    .adaptiveRefreshTriggersTimeout(30, TimeUnit.SECONDS)
                                    .build();
  
    clusterClient.setOptions(ClusterClientOptions.builder()
                                    .topologyRefreshOptions(topologyRefreshOptions)
                                    .build());
    ...
  
    clusterClient.shutdown();
  

• Modify the following code to configure and connect the redis cluster client in Spring boot2.x:

    @Configuration
    public class LettuceRedisConfig {
  
        @Autowired
        private RedisProperties redisProperties;
  
        [@Bean](https://my.oschina.net/bean)
        public RedisTemplate redisTemplate(@Qualifier("lettuceConnectionFactoryUvPv") RedisConnectionFactory lettuceConnectionFactoryUvPv) {
            RedisTemplate<String, Object> template = new RedisTemplate<String, Object>();
            template.setConnectionFactory(lettuceConnectionFactoryUvPv);
            Jackson2JsonRedisSerializer jackson2JsonRedisSerializer =
                    new Jackson2JsonRedisSerializer(Object.class);
            ObjectMapper om = new ObjectMapper();
            om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
            jackson2JsonRedisSerializer.setObjectMapper(om);
            StringRedisSerializer stringRedisSerializer = new StringRedisSerializer();
            template.setKeySerializer(stringRedisSerializer);
            template.setHashKeySerializer(stringRedisSerializer);
            template.setValueSerializer(jackson2JsonRedisSerializer);
            template.setHashValueSerializer(jackson2JsonRedisSerializer);
            template.afterPropertiesSet();
            return template;
        }
  
  
        @Bean(destroyMethod = "destroy")
        public LettuceConnectionFactory lettuceConnectionFactoryUvPv(){
  
            List<String> clusterNodes = redisProperties.getCluster().getNodes();
            Set<RedisNode> nodes = new HashSet<RedisNode>();
            clusterNodes.forEach(address -> nodes.add(new RedisNode(address.split(":")[0].trim(),
                    Integer.parseInt(address.split(":")[1]))));
            RedisClusterConfiguration clusterConfiguration = new RedisClusterConfiguration();
            clusterConfiguration.setClusterNodes(nodes);
            clusterConfiguration.setPassword(RedisPassword.of(redisProperties.getPassword()));
            clusterConfiguration.setMaxRedirects(redisProperties.getCluster().getMaxRedirects());
  
            GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
            poolConfig.setMaxIdle(redisProperties.getLettuce().getPool().getMaxIdle());
            poolConfig.setMinIdle(redisProperties.getLettuce().getPool().getMinIdle());
            poolConfig.setMaxTotal(redisProperties.getLettuce().getPool().getMaxActive());
  
            return new LettuceConnectionFactory(clusterConfiguration,
                    getLettuceClientConfiguration(poolConfig));
        }
  
        private LettuceClientConfiguration getLettuceClientConfiguration(GenericObjectPoolConfig genericObjectPoolConfig) {
            ClusterTopologyRefreshOptions topologyRefreshOptions =
                    ClusterTopologyRefreshOptions.builder()
                    .enableAllAdaptiveRefreshTriggers()
                    .adaptiveRefreshTriggersTimeout(Duration.ofSeconds(25)) 
                    .enablePeriodicRefresh(Duration.ofSeconds(20))  
                    .build();
            return LettucePoolingClientConfiguration.builder()
                    .poolConfig(genericObjectPoolConfig)
                    .clientOptions(ClusterClientOptions.builder().topologyRefreshOptions(topologyRefreshOptions).build())
                    .build();
        }
    }
  

In the above code, you need to set automatic refresh to connect the connection pool to the new master node, so as to avoid the problem that you don't need to replace it with jedis. Some people may say that replacing lettuce directly with jedis will reduce the efficiency. lettuce is also officially recommended.