colony

Redis still holds all the data in the cluster in master-slave replication mode, which limits the total data storage of the cluster to the server with the smallest memory.

Additionally, scaling out Redis horizontally is cumbersome, and client fragmentation is often used to solve this problem, with the client deciding which node each key is stored in.However, if you want to add new nodes during capacity expansion, you need to migrate the data manually. To ensure data consistency during the migration process, you also need to temporarily offline Redis.

Redis introduced cluster mode in version 3.0, in which Redis determines which node key s are stored and from which nodes to query.

Prepare 8 virtual machines and install redis:

• 192.168.2.104
• 192.168.2.105
• 192.168.2.106
• 192.168.2.107
• 192.168.2.108
• 192.168.2.109
• 192.168.2.110
• 192.168.2.111

The first six nodes form a cluster, and then the last two nodes are added to the cluster in an expanded form.

To configure

To use a cluster, first turn on the configuration of the cluster:

cluster-enabled yes

Open port 16937 if firewalled (default)

After Redis is started, perform INFO cluster queries under the redis-cli terminal to see if the cluster is healthy:

[root@localhost redis-5.0.7]# redis-cli
127.0.0.1:6379> INFO cluster
# Cluster
cluster_enabled:1

I tested with six nodes:

Next, execute the redis cluster command.

Execute commands on each server node:

redis-cli --cluster create 192.168.2.104:6379 192.168.2.105:6379 192.168.2.106:6379 192.168.2.107:6379 192.168.2.108:6379 192.168.2.109:6379 --cluster-replicas 1

Fill in the ip and port of the six nodes, and the parameter--cluster-replicas 1 is the number of secondary databases each database has.

After executing this command, there are three master databases and three slave databases in the cluster, so your cluster has been set up.The CLUSTER NODES command can be executed under the redis-cli terminal to query the node state of the cluster.

Slot allocation

Between understanding the slots, first note how Redis partitioned Key.

Redis calculates a value from CRC16 (Key)%16384 and stores the data in the slot corresponding to the value.

There are 16384 slots in Redis Cluster mode. As to why there are 16384 slots, you can check the reason on the Internet and leave it alone.

So in a cluster, all keys are assigned to one of the 16384 slots.Then the three master databases default to 16384/3 slots.

The CLUSTER SLOTS command can be executed under the redis-cli terminal to see the slot intervals responsible for each node, and Redis does not require that the slots responsible for each primary node be continuous.

Expansion

When adding new nodes to a cluster, the node types are divided into primary and secondary nodes.

Add Primary Node

Add the primary node by executing the command as follows:

redis-cli --cluster add-node 192.168.2.110:6379 192.168.2.106:6379

The last two parameters of the command are the IP and port of the new node (192.168.2.110:6379) on the left and the IP and port of any node in the cluster (192.168.2.106:6379) on the right.This command adds a new node to the cluster and sets it as the primary node.

The primary node that just joined the cluster has no responsible slot. Execute the command:

redis-cli --cluster reshard 192.168.2.106:6379

This command is used to request a slot, and the last parameter is the IP and port of any node in the cluster.There are several interactive steps to execute the command:

Terminal: How many slots do you want to move (from 1 to 16384)?

Fill in a number that represents how many slot s each party has applied for to a new node added to the group. I wrote 4096 and then asked again:

what is the receiving node ID?

Request to enter a nodeID to receive these 4096 slot s, I queried the CLUSTER NODES command to fill in the 192.168.2.110 odeID, and then asked:

Please enter all the source node IDs.

  Type 'all' to use all the nodes as source nodes for the hash slots.

  Type 'done' once you entered all the source nodes IDs.

There are two ways to require filling in slots that move those nodes to a new node:

One is to write the nodeID of the specified node, and then enter done.

One is to write all, and Redis automatically extracts 4096 slot s from all nodes to the new node.

I write all here, return, enter yes to confirm execution.The primary node has added and applied for slot s.

Add Slave Node

The following adds a slave node to the primary node just added (192.168.2.110) to execute the command:

redis-cli --cluster add-node --cluster-slave --cluster-master-id 691b4cb6563da921d4d9bd191d8fa98a7cef0c23 192.168.2.111:6379 192.168.2.110:6379

The last three parameters are the NodeID of the primary node, the IP and port of the secondary node, and the IP and port of the primary node.

This completes an expansion, adds a primary node and its corresponding slave nodes, and assigns 4096 slot s to the new node, which can be queried by CLUSTER NODES and CLUSSTER SLOTS:

[root@localhost redis-5.0.7]# redis-cli
127.0.0.1:6379> CLUSTER NODES
691b4cb6563da921d4d9bd191d8fa98a7cef0c23 192.168.2.110:6379@16379 master - 0 1580832752462 9 connected 0-1364 5461-6826 10923-12287
ee1a9c31eb1571254e2679e5e84a32f7931c8b37 192.168.2.107:6379@16379 slave 3fab72ba46dd8f50fc4c665f1011c41940152385 0 1580832753000 8 connected
392006bd13d86e92d8c5dca2d89030adb7d2ed51 192.168.2.109:6379@16379 slave 435eff563051e89e7680dbc8a6146c7f4b637a36 0 1580832750445 6 connected
931f465e35513355c7043b2faba7974d1c762020 192.168.2.108:6379@16379 slave 76130efb66918698579473a592a5b9188922fa73 0 1580832749000 5 connected
435eff563051e89e7680dbc8a6146c7f4b637a36 192.168.2.105:6379@16379 master - 0 1580832751000 2 connected 6827-10922
76130efb66918698579473a592a5b9188922fa73 192.168.2.104:6379@16379 myself,master - 0 1580832751000 1 connected 1365-5460
3fab72ba46dd8f50fc4c665f1011c41940152385 192.168.2.106:6379@16379 master - 0 1580832753468 8 connected 12288-16383
f1e3cd10ed6ef51a04601f179b8446c58f2f52f1 192.168.2.111:6379@16379 slave 691b4cb6563da921d4d9bd191d8fa98a7cef0c23 0 1580832751453 9 connected

[root@localhost redis-5.0.7]# redis-cli
127.0.0.1:6379> CLUSTER SLOTS
1) 1) (integer) 0
   2) (integer) 1364
   3) 1) "192.168.2.110"
      2) (integer) 6379
      3) "691b4cb6563da921d4d9bd191d8fa98a7cef0c23"
   4) 1) "192.168.2.111"
      2) (integer) 6379
      3) "f1e3cd10ed6ef51a04601f179b8446c58f2f52f1"
2) 1) (integer) 5461
   2) (integer) 6826
   3) 1) "192.168.2.110"
      2) (integer) 6379
      3) "691b4cb6563da921d4d9bd191d8fa98a7cef0c23"
   4) 1) "192.168.2.111"
      2) (integer) 6379
      3) "f1e3cd10ed6ef51a04601f179b8446c58f2f52f1"
3) 1) (integer) 10923
   2) (integer) 12287
   3) 1) "192.168.2.110"
      2) (integer) 6379
      3) "691b4cb6563da921d4d9bd191d8fa98a7cef0c23"
   4) 1) "192.168.2.111"
      2) (integer) 6379
      3) "f1e3cd10ed6ef51a04601f179b8446c58f2f52f1"
4) 1) (integer) 6827
   2) (integer) 10922
   3) 1) "192.168.2.105"
      2) (integer) 6379
      3) "435eff563051e89e7680dbc8a6146c7f4b637a36"
   4) 1) "192.168.2.109"
      2) (integer) 6379
      3) "392006bd13d86e92d8c5dca2d89030adb7d2ed51"
5) 1) (integer) 1365
   2) (integer) 5460
   3) 1) "192.168.2.104"
      2) (integer) 6379
      3) "76130efb66918698579473a592a5b9188922fa73"
   4) 1) "192.168.2.108"
      2) (integer) 6379
      3) "931f465e35513355c7043b2faba7974d1c762020"
6) 1) (integer) 12288
   2) (integer) 16383
   3) 1) "192.168.2.106"
      2) (integer) 6379
      3) "3fab72ba46dd8f50fc4c665f1011c41940152385"
   4) 1) "192.168.2.107"
      2) (integer) 6379
      3) "ee1a9c31eb1571254e2679e5e84a32f7931c8b37"

Test Cluster

When writing data to a cluster via redis-cli, the following errors may occur:

[root@localhost redis-5.0.7]# redis-cli
127.0.0.1:6379> set hello world
(error) MOVED 866 192.168.2.110:6379

I executed the set hello world command on the redis-cli terminal on 192.168.2.104, prompting for error, because the key equals 866 after CRC16 ('hello')%16384, and the node where slot 866 is located is not the current node.You can start the terminal with redis-cli-c and test it.

[root@localhost redis-5.0.7]# redis-cli -c
127.0.0.1:6379> set hello world!!
-> Redirected to slot [866] located at 192.168.2.110:6379
OK

Failure Recovery

Primary node has no slave node

If your cluster node has a primary node down and it is not a secondary node, your cluster is not available by default.

If you want the cluster to use a cluster in this case, you can modify the configuration:

cluster-require-full-coverage no # The default is yes

This allows normal use.

Primary node has slave nodes

If the primary node is down, but there is at least one slave node, then the cluster will fail over, upgrade one of the downloaded primary nodes to the primary node, and select which slave node is based on the Raft algorithm as the algorithm for selecting the leading sentry from the sentry.

When one of them is selected from the database, it upgrades itself to the master database through SLAVEOF ON ONE, and converts the slot from the down-dropped master database to itself.

As a test, the reids service was shut down at 192.168.2.110:

[root@localhost redis-5.0.7]# /etc/init.d/redis stop
Stopping ...
Redis stopped

Looking at the cluster information at other nodes, you can see that 192.168.2.111 has been upgraded to master and is responsible for slots originally owned by 192.168.2.110:

127.0.0.1:6379> cluster nodes
691b4cb6563da921d4d9bd191d8fa98a7cef0c23 192.168.2.110:6379@16379 master,fail - 1580833747361 1580833745000 9 disconnected
ee1a9c31eb1571254e2679e5e84a32f7931c8b37 192.168.2.107:6379@16379 slave 3fab72ba46dd8f50fc4c665f1011c41940152385 0 1580833864374 8 connected
392006bd13d86e92d8c5dca2d89030adb7d2ed51 192.168.2.109:6379@16379 slave 435eff563051e89e7680dbc8a6146c7f4b637a36 0 1580833861000 6 connected
931f465e35513355c7043b2faba7974d1c762020 192.168.2.108:6379@16379 slave 76130efb66918698579473a592a5b9188922fa73 0 1580833865382 5 connected
435eff563051e89e7680dbc8a6146c7f4b637a36 192.168.2.105:6379@16379 master - 0 1580833862000 2 connected 6827-10922
76130efb66918698579473a592a5b9188922fa73 192.168.2.104:6379@16379 myself,master - 0 1580833864000 1 connected 1365-5460
3fab72ba46dd8f50fc4c665f1011c41940152385 192.168.2.106:6379@16379 master - 0 1580833862356 8 connected 12288-16383
f1e3cd10ed6ef51a04601f179b8446c58f2f52f1 192.168.2.111:6379@16379 master - 0 1580833863365 11 connected 0-1364 5461-6826 10923-12287

The following is the redis service starting at 192.168.2.110:

[root@localhost redis-5.0.7]# /etc/init.d/redis start
Starting Redis server...
2706:C 05 Feb 2020 00:32:59.512 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2706:C 05 Feb 2020 00:32:59.512 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=2706, just started
2706:C 05 Feb 2020 00:32:59.512 # Configuration loaded

Looking at the cluster information, 192.168.2.110 has become a slave node of 192.168.2.111:

127.0.0.1:6379> cluster nodes
691b4cb6563da921d4d9bd191d8fa98a7cef0c23 192.168.2.110:6379@16379 slave f1e3cd10ed6ef51a04601f179b8446c58f2f52f1 0 1580834002000 11 connected
ee1a9c31eb1571254e2679e5e84a32f7931c8b37 192.168.2.107:6379@16379 slave 3fab72ba46dd8f50fc4c665f1011c41940152385 0 1580834000000 8 connected
392006bd13d86e92d8c5dca2d89030adb7d2ed51 192.168.2.109:6379@16379 slave 435eff563051e89e7680dbc8a6146c7f4b637a36 0 1580834000000 6 connected
931f465e35513355c7043b2faba7974d1c762020 192.168.2.108:6379@16379 slave 76130efb66918698579473a592a5b9188922fa73 0 1580834001773 5 connected
435eff563051e89e7680dbc8a6146c7f4b637a36 192.168.2.105:6379@16379 master - 0 1580834002780 2 connected 6827-10922
76130efb66918698579473a592a5b9188922fa73 192.168.2.104:6379@16379 myself,master - 0 1580833999000 1 connected 1365-5460
3fab72ba46dd8f50fc4c665f1011c41940152385 192.168.2.106:6379@16379 master - 0 1580834000000 8 connected 12288-16383
f1e3cd10ed6ef51a04601f179b8446c58f2f52f1 192.168.2.111:6379@16379 master - 0 1580834000764 11 connected 0-1364 5461-6826 10923-12287