This article is shared from Huawei cloud community Evaluation of ink Sky Wheel: the impact of GaussDB(for Redis) big Key operation , author: Goss Redis official blog.
In the previous article Evaluation of ink Sky Wheel: stability and capacity expansion performance of GaussDB(for Redis) In, we use the multithreaded pressure measurement tool memtier_benchmark conducted a comparative pressure test between Huawei GaussDB(for Redis) and native Redis, and found that native Redis is prone to OOM failure, and the capacity expansion operation will be very slow, which will bring great pressure to the operation and maintenance. In contrast, Huawei GaussDB(for Redis) not only has stable performance, but also has the ability to expand capacity in seconds during the voltage test. The expansion operation has no impact on business reading and writing. Huawei GaussDB(for Redis) supports full data dropping. The basic component service of GaussDB provides redundant storage of three copies of underlying data, which can ensure zero data loss. Huawei GaussDB(for Redis) is a suitable model if the usage scenario not only meets the high performance of KV query, but also hopes that the data can be valued and not lost.
Most of us have encountered the serious impact of big key on Redis performance in the actual production environment. Next, let's test the performance of GaussDB(for Redis) as a "performance killer" of big key through several simple experiments, and what are the performance improvements compared with native Redis?
1. Access big key
First, create a large hash key in GaussDB(for Redis) and native redis respectively.
Edit a simple lua script and insert 10 million pieces of data into a hash key.
# vim redis-bigkey.lua local result for var=1,10000000,1 do redis.call('hset',KEYS[1],var,var) redis.call('lpush',KEYS[2],var) redis.call('sadd',KEYS[3],var) end return result
Insert a large key into GaussDB(for Redis) [192.168.0.226:8635]
# redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 --eval /root/redis-bigkey.lua 3 hset_test list_test set_test (nil) # redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 hlen hset_test (integer) 10000000 # redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 scard sadd_test (integer) 10000000 # redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 llen lpush_test (integer) 10000000
Insert a large key into the native Redis [192.168.0.135]
# redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 --eval /root/redis-bigkey.lua 3 hset_test list_test set_test (nil) # redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 hlen hset_test (integer) 10000000 # redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 scard sadd_test (integer) 10000000 # redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 llen lpush_test (integer) 10000000
Using memtier_benchmark simulates the business pressure and accesses the big key to observe the impact on the business qps.
Edit a simple shell script to frequently access large key s
#!/bin/bash while true do redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 hget hset_test $RANDOM redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 hget hset_test $RANDOM redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 LREM lpush_test 0 $RANDOM redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 LREM lpush_test 0 $RANDOM redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 SISMEMBER sadd_test $RANDOM redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 SISMEMBER sadd_test $RANDOM done
Using memtier_benchmark for pressure measurement, reading and writing mixed scenes. From the feedback performance data, we can see that the number of ops/sec per second of GaussDB(for Redis) and Redis are 140000 and 130000 respectively, with little difference.
#GaussDB(for Redis) memtier_benchmark -s 192.168.0.226 -a XXXXXXX -p 8635 -c 16 -t 12 -n 100000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000000 --key-minimum=1 --key-prefix= --ratio=1:1 --out-file=./result_gauss_setget.log [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 3%, 4 secs] 12 threads: 582045 ops, 144053 (avg: 145472) ops/sec, 21.16MB/sec (avg: 21.51MB/sec), 1.33 (avg: 1.32) msec latency #Native Redis memtier_benchmark -s 192.168.0.135 -a XXXXXXX -p 6379 -c 16 -t 12 -n 100000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000000 --key-minimum=1 --key-prefix= --ratio=1:1 --out-file=./result_redis_setget.log [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 7%, 11 secs] 12 threads: 1430798 ops, 132637 (avg: 130051) ops/sec, 70.51MB/sec (avg: 68.79MB/sec), 1.44 (avg: 1.47) msec latency
After starting the shell script and observing again, it is found that the number of operations per second of GaussDB(for Redis) has almost no change, while the number of operations per second of native Redis fluctuates greatly, even reduced to about 3k. It shows that the large key operation has a great impact on the performance of native Redis, and the impact on GaussDB(for Redis) is controllable.
# bash hget_bigkey.sh #GaussDB(for Redis) # memtier_benchmark -s 192.168.0.226 -a XXXXXXX -p 8635 -c 16 -t 12 -n 100000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000000 --key-minimum=1 --key-prefix= --ratio=1:1 --out-file=./result_gauss_setget.log [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 47%, 64 secs] 12 threads: 9099444 ops, 139186 (avg: 142163) ops/sec, 20.60MB/sec (avg: 20.96MB/sec), 1.38 (avg: 1.35) msec latency #Native Redis # memtier_benchmark -s 192.168.0.135 -a XXXXXXX -p 6379 -c 16 -t 12 -n 100000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000000 --key-minimum=1 --key-prefix= --ratio=1:1 --out-file=./result_redis_setget.log [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 29%, 75 secs] 12 threads: 5607700 ops, 3329 (avg: 74759) ops/sec, 1.80MB/sec (avg: 40.08MB/sec), 52.35 (avg: 2.55) msec latencyy
2. Delete large key
Continue to use memtier_benchmark tests GaussDB(for Redis) and native Redis in a read-only scenario. Deleting large key s in GaussDB(for Redis) can be completed quickly and has little impact on performance.
#GaussDB(for Redis) # memtier_benchmark -s 192.168.0.226 -a XXXXXXX -p 8635 -c 16 -t 12 -n 100000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000000 --key-minimum=1 --key-prefix= --ratio=0:1 --out-file=./result_gauss_setget.log Writing results to ./result_gauss_setget.log... [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 4%, 5 secs] 12 threads: 719216 ops, 151326 (avg: 143795) ops/sec, 22.16MB/sec (avg: 21.13MB/sec), 1.27 (avg: 1.33) msec latency # time redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 del sadd_test (integer) 1 real 0m0.003s user 0m0.001s sys 0m0.002s # memtier_benchmark -s 192.168.0.226 -a XXXXXXX -p 8635 -c 16 -t 12 -n 100000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000000 --key-minimum=1 --key-prefix= --ratio=0:1 --out-file=./result_gauss_setget.log Writing results to ./result_gauss_setget.log... [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 42%, 57 secs] 12 threads: 8031731 ops, 144874 (avg: 140890) ops/sec, 21.46MB/sec (avg: 20.77MB/sec), 1.32 (avg: 1.36) msec latency
In contrast to native Redis, deleting large keys takes 3 seconds and has a great impact on performance during deletion. It can be observed that ops/sec becomes 0 during deletion, that is, there is no way to respond normally during large key deletion.
#Native Redis # memtier_benchmark -s 192.168.0.135 -a XXXXXXX -p 6379 -c 16 -t 12 -n 100000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000000 --key-minimum=1 --key-prefix= --ratio=0:1 --out-file=./result_redis_setget.log Writing results to ./result_redis_setget.log... [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 6%, 7 secs] 12 threads: 1107132 ops, 157621 (avg: 158125) ops/sec, 16.07MB/sec (avg: 16.13MB/sec), 1.22 (avg: 1.21) msec latency # time redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 del sadd_test (integer) 1 real 0m3.001s user 0m0.000s sys 0m0.003s # memtier_benchmark -s 192.168.0.135 -a XXXXXXX -p 6379 -c 16 -t 12 -n 100000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000000 --key-minimum=1 --key-prefix= --ratio=0:1 --out-file=./result_redis_setget.log Writing results to ./result_redis_setget.log... [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 57%, 68 secs] 12 threads: 1015893 ops, 0 (avg: 126961) ops/sec, 0.00KB/sec (avg: 12.98MB/sec), -nan (avg: 1.13) msec latencyy
Manually deleting large keys has a significant impact on performance. If you set the expiration time of large keys and let Redis delete expired data, will it have a performance impact? Here's a simple test
Manually set the expiration time of the large key and start memtier_benchmark read-write hybrid test to see the impact on performance. Through the test, it is found that the expiration of large keys has little impact on the performance of GaussDB(for Redis).
#GaussDB(for Redis) [root@ecs-ef13-0001 ~]# redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 EXPIRE sadd_test 8 && redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 EXPIRE sadd_test1 12 (integer) 1 (integer) 1 [root@ecs-ef13-0001 ~]# memtier_benchmark -s 192.168.0.226 -a XXXXXXX -p 8635 -c 16 -t 12 -n 10000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000 --key-minimum=1 --key-prefix= --ratio=1:1 --out-file=./result_gauss_setget.log Writing results to ./result_gauss_setget.log... [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 100%, 17 secs] 0 threads: 1920000 ops, 106367 (avg: 109940) ops/sec, 105.02MB/sec (avg: 108.55MB/sec), 1.74 (avg: 1.74) msec latency
When testing the native Redis, we found that the large key expiration operation almost blocked the normal reading and writing. In memtier_ During the benchmark test, the ops/sec index is 0. It will return to normal only after the expiration of the large key.
#Native Redis [root@ecs-ef13-0001 ~]# redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 EXPIRE sadd_test 8 && redis-cli -h 192.168.0.135 -a XXXXXXX -p 6379 EXPIRE sadd_test1 12 (integer) 1 (integer) 1
[root@ecs-ef13-0001 ~]# memtier_benchmark -s 192.168.0.135 -a XXXXXXX -p 6379 -c 16 -t 12 -n 10000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000 --key-minimum=1 --key-prefix= --ratio=1:1 --out-file=./result_redis_setget.log Writing results to ./result_redis_setget.log... [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 100%, 42 secs] 0 threads: 1920000 ops, 134502 (avg: 45551) ops/sec, 132.80MB/sec (avg: 44.98MB/sec), 1.43 (avg: 4.21) msec latency
Although the expiration of open source redis also supports asynchrony, users need to manually configure the policy; For deletion, UNLINK is required to replace the conventional DEL. The specific impact on performance may be reduced. No in-depth verification will be conducted this time. The deletion and expiration of Huawei GaussDB(for Redis) have an impact on performance 0.
3. Influence of big key on capacity expansion of GaussDB(for Redis)
In the previous article, we tested the online capacity expansion function of GaussDB(for Redis). After testing, GaussDB(for Redis) can realize second level capacity expansion without affecting business reading and writing. This time, we will add some "difficulties" to see whether the capacity expansion operation of GaussDB(for Redis) can achieve second level and zero service perception in the case of large key s.
Insert a large key into GaussDB(for Redis) in advance
[root@ecs-ef13-0001 ~]# redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 scard sadd_test4 (integer) 10000000 [root@ecs-ef13-0001 ~]# redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 scard sadd_test5 (integer) 10000000 [root@ecs-ef13-0001 ~]# redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 scard sadd_test6 (integer) 10000000 [root@ecs-ef13-0001 ~]# redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 scard sadd_test7 (integer) 10000000 [root@ecs-ef13-0001 ~]# redis-cli -h 192.168.0.226 -a XXXXXXX -p 8635 scard sadd_test8 (integer) 10000000
Using memtier_benchmark simulates read-write requests and performs capacity expansion on the console at the same time. Like the previous test results, GaussDB(for Redis) also realizes the second scale expansion of zero perception of business
[root@ecs-ef13-0001 ~]# memtier_benchmark -s 192.168.0.226 -a XXXXXXX -p 8635 -c 16 -t 12 -n 50000 --random-data --randomize --distinct-client-seed -d 1000 --key-maximum=65000 --key-minimum=1 --key-prefix= --ratio=0:1 --out-file=./result_gauss_setget.log Writing results to ./result_gauss_setget.log... [RUN #1] Preparing benchmark client... [RUN #1] Launching threads now... [RUN #1 9%, 20 secs] 12 threads: 902361 ops, 42634 (avg: 45112) ops/sec, 41.99MB/sec (avg: 44.44MB/sec), 4.53 (avg: 4.23) msec latencycy
4. Summary
The access, deletion and other operations of native Redis to large keys will seriously block the normal access of services, which is determined by Redis's own architecture of single thread processing requests. When using native Redis, it is necessary to strictly restrict the use of large keys. Once a large key appears, it usually has a "fatal" impact on the performance of the system.
On the contrary, GaussDB(for Redis) adopts multi-threaded architecture. The impact on performance of accessing and deleting large keys and capacity expansion in the presence of large keys is controllable. 1) In the big key access scenario, because GaussDB(for Redis) adopts a multi-threaded architecture, it is not easy to block other business operations. 2) In the scenario of large key deletion, due to the different logic implemented by GaussDB(for Redis), the deletion operation can be completed quickly without affecting the business. 3) In the capacity expansion scenario, GaussDB(for Redis) does not involve key migration, and large keys have no impact on capacity expansion.
To sum up, although it is generally recommended to avoid large keys in business design, Huawei cloud GaussDB(for Redis) performs better in some business scenarios where a small number of large keys need to be operated.
In addition, from the perspective of business development, when multiple businesses share one instance, if GaussDB(for Redis) is used, even if other businesses introduce large key s, their own businesses will not be greatly affected.
Click follow to learn about Huawei's new cloud technology for the first time~