Although Redis itself also provides redis-benchmark tools to test Redis's performance, in order to customize the test dimension freely, we still test it by writing our own scripts.
Redis's official documentation also mentions that simply starting a loop and then sending operation commands to Redis in the loop is not a performance test for Redis, but a test for network latency. To really test the concurrency performance of Redis, you need to use multiple Redis connections or use pipelining to aggregate multiple commands. Of course, you can also use multi-threading or multi-process.
We use the Python script as a test tool for Redis and use a Redis connection scheme for each process.
The test script is as follows:
#!/usr/bin/env python
# -*- coding:utf-8 -*-
import time
import datetime
from multiprocessing import Process
import redis
def write_worker(n):
print 'in write worker: %d, time: %s' % (n, datetime.datetime.now())
r = redis.StrictRedis(host='10.88.115.114', port=6380)
s = 'a' * 1024 * 1024 * 10
t1 = time.time()
bigkey = 'bigkey%d' % (n + 100)
r.set(bigkey, s)
t2 = time.time()
print 'write time: %f' % (t2 - t1)
def read_worker(n):
print 'in read worker: %d, time: %s' % (n, datetime.datetime.now())
r = redis.StrictRedis(host='10.88.115.114', port=6380)
t1 = time.time()
bigkey = 'bigkey%d' % n
r.get(bigkey)
t2 = time.time()
print 'read time: %f' % (t2 - t1)
def test():
read_processes = []
for i in range(0, 100):
read = Process(target=read_worker, args=(i,))
read.start()
read_processes.append(read)
write_processes = []
for i in range(0, 100):
write = Process(target=write_worker, args=(i,))
write.start()
write_processes.append(write)
for write in write_processes:
write.join()
for read in read_processes:
read.join()
print 'end press test'
if __name__ == '__main__':
test()redis-py's trictRedis already provides connection pool functionality that can be used directly in multi-threading, where we use a multi-process approach and use a separate StrictRedis instance for each process.
In the code above, we started a lot of reading and writing processes. In Redis write operation, one SET request is sent at a time, and the size of the data written is 100 MB. In read operation, the data of GET 100 MB is read at a time (the key data of GET request needs to be prepared in advance). The purpose of the test program is to test Redis's performance in reading and writing large data when dealing with high concurrency.
The test results can not be posted here. Interested readers can also run this Python script to get the test results. From the test results, Redis's performance is excellent.
Here we highlight the problem that Redis services will report errors during testing.
After writing a large amount of data (100 10MB, or 1GB), the Redis process begins to appear the following logs:
[2418] 21 Apr 09:38:24.041 * 10 changes in 300 seconds. Saving...
[2418] 21 Apr 09:38:24.042 # Can't save in background: fork: Cannot allocate memory
Client execution of write command will result in an error:
127.0.0.1:6380> set key value
(error) MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
Looking up Redis information on the Internet shows that this is because Redis has opened RDB functions. RDB is used to save Redis data snapshots and persist data snapshots to hard disks. The following is Redis's default RDB configuration:
save 900 1 save 300 10 save 60 10000
It means that there is one modification in 900 seconds, 10 modifications in 300 seconds and 10,000 modifications in 60 seconds, which will persist the data to disk.
To solve the problem of Redis error, you can turn off RDB function or optimize Linux memory configuration. Specific reference "REDIS Memory Optimization under LINUX".
sysctl vm.overcommit_memory=1After the command is executed with root privilege, the Redis service is restarted, and then the test program is executed, so that the problem of Redis error no longer occurs.
It should be pointed out that according to many articles on the Internet, the problem of error can be solved by modifying the stop-writes-on-bgsave-error option in redis.conf configuration, which is actually just to cover up the problem. Because the following configuration is used:
stop-writes-on-bgsave-error no
Although RDB cannot save Redis data snapshots, Redis can continue to accept new Redis requests.
In order to fundamentally solve the problem of RDB error, it is still necessary to modify the parameters of the vm. overcommit_memory Linux kernel.