Redis provides a list, hash, zset and other complex types of data structures. When business is used, a key may be too large due to unreasonable key design. Because redis is a simple single-threaded model, business will have a certain impact on the acquisition or deletion of large keys. In addition, in cluster mode, the generation of large keys can easily lead to a full memory of a sub-node. We need to provide search tools for large keys.

Initialization environment

Install python client

Download python client
wget "https://pypi.python.org/packages/68/44/5efe9e98ad83ef5b742ce62a15bea609ed5a0d1caf35b79257ddb324031a/redis-2.10.5.tar.gz#md5=3b26c2b9703b4b56b30a1ad508e31083"

Decompression installation

tar -xvf redis-2.10.5.tar.gz
cd redis-2.10.5
sudo python setup.py install

Scanning script

Traversal key

The master-slave version of Redis can be scanned by scan command, and the cluster version can be scanned by ISCAN command. The command rules are as follows, in which the number of node s can be obtained by info command.

ISCAN idx cursor [MATCH pattern] [COUNT count] (idx is the id of the node, starting from 0, 16 to 64 GB cluster instances are 8 nodes, so IDX is 0 to 7, 128G 256 GB is 16 nodes)

Scanning script

import sys
import redis

def check_big_key(r, k,node):
  bigKey = False
  length = 0 
  try:
    type = r.type(k)
    if type == "string":
      length = r.strlen(k)
    elif type == "hash":
      length = r.hlen(k)
    elif type == "list":
      length = r.llen(k)
    elif type == "set":
      length = r.scard(k)
    elif type == "zset":
      length = r.zcard(k)
  except:
    return
  if length > 10240:
    bigKey = True
  if bigKey :
    print db,k,type,length,node

def find_big_key_normal(db_host, db_port, db_password, db_num):
  r = redis.StrictRedis(host=db_host, port=db_port, password=db_password, db=db_num)
  for k in r.scan_iter(count=1000):
    check_big_key(r, k,0)

def find_big_key_sharding(db_host, db_port, db_password, db_num, nodecount):
  r = redis.StrictRedis(host=db_host, port=db_port, password=db_password, db=db_num)
  cursor = 0
  for node in range(0, nodecount) :
    while True:
      iscan = r.execute_command("iscan",str(node), str(cursor), "count", "1000")
      for k in iscan[1]:
        check_big_key(r, k,node)
      cursor = iscan[0]
      if cursor == "0":
        break;

if __name__ == '__main__':
  if len(sys.argv) != 4:
     print 'Usage: python ', sys.argv[0], ' host port password '
     exit(1)
  db_host = sys.argv[1]
  db_port = sys.argv[2]
  db_password = sys.argv[3]
  r = redis.StrictRedis(host=db_host, port=int(db_port), password=db_password)
  nodecount = r.info()['nodecount']
  keyspace_info = r.info("keyspace")
  for db in keyspace_info:
    print 'check ', db, ' ', keyspace_info[db]
    if nodecount > 1:
      find_big_key_sharding(db_host, db_port, db_password, db.replace("db",""), nodecount)
    else:
      find_big_key_normal(db_host, db_port, db_password, db.replace("db", ""))

It can be executed by Python find_bigkey host 6379 password to support large key lookup of master-slave and cluster versions of Redis in Aliyun. The default threshold of large key is 10240, that is to say, the value of string type is greater than 10240, the list length is greater than 10240, and the number of field s is greater than 10240 for list type. Big key. In addition, by default, the script searches 1000 keys at a time, which has a relatively low impact on the business, but it is better to operate at the peak of the business to avoid the impact of scan commands on the business. The number of script output is as follows, and the last one is the serial number of the nodes in the cluster.

check  db0   {'keys': 79648, 'expires': 0, 'avg_ttl': 0}
db0 mylist list 59819 2