Nosql overview

Why use Nosql

Stand alone MySQL Era
In the 1990s, the number of visits to a website was generally not too large, and a single database was sufficient. However, with the increase of users, the following problems occurred:

• The amount of data is too large for one machine
• The index of data (B+ tree) can't fit into a machine's memory
• The number of visits is too large (mixed reading and writing), and one machine can't bear it

Memcached + MySQL + vertical split
80% of the website is reading. It's very troublesome to query the database every time! Therefore, we hope to reduce the pressure on the database. We can use cache to ensure efficiency!

Sub database and sub table + horizontal split + MySQL Cluster

Current basic Internet projects

User's personal information, social networks, geographical location. User generated data, user logs and so on are growing explosively!
At this time, we need to use NoSQL database. NoSQL can handle the above situations well!

What is Nosql

NoSQL = Not Only SQL (not just SQL)
Relational database: Table: column + row. The data structure under the same table is the same
Non relational database: data storage has no fixed format and can be expanded horizontally
Nosql features

1. Easy to expand (there is no relationship between data, so it is easy to expand)
2. Large amount of data and high performance (Redis can write 80000 times and read 110000 times a second. NoSQL's cache record level is a fine-grained cache, and its performance will be relatively high!)
3. Data types are diverse! (there is no need to design the database in advance, and it can be used at any time)

Getting started with Redis

What is Redis

Redis (Remote Dictionary Server), i.e. remote Dictionary Service
It is an open source log and key value database written in ANSI C language, supporting network, memory based and persistent, and provides API s in multiple languages
The data is cached in memory. redis will periodically write the updated data to disk or write the modification operation to the additional record file. On this basis, master-slave synchronization is realized
Redis is an open source (BSD licensed) data structure server with in memory storage, which can be used as database, cache and message queue agent. It supports string, hash table, list, set, ordered set, bitmap, hyperlogs and other data types. Built in replication, Lua script, LRU retraction, transaction and disk persistence at different levels. At the same time, it provides high availability through Redis Sentinel and automatic partition through Redis Cluster

characteristic

1. Diverse data types

2. Persistence

3. colony

4. affair
   ...

Single thread

Redis is single threaded!, Redis puts all data in memory, so using single thread to operate is the most efficient. Multithreading (CPU context will switch: time-consuming operation!), For the memory system, if there is no context switching, the efficiency is the highest. Multiple reads and writes are on one CPU. In the case of storing data in memory, single thread is the best solution

Installation under Linux

1. Download the installation package redis-6.2.6 tar. gz

2. Upload to Linux, / usr/local/software Directory

3. Enter / usr/local/software directory and unzip tar -zxvf redis-6.2.6 tar. gz

4. Enter the decompression file cd redis-6.2.6

5. Installation environment yum install gcc-c++

6. Execute make and then make install

7. The default installation path of redis is / usr/local/bin. Enter this directory and copy the redis configuration file to / usr/local/bin / rainhey_ config[ root@node1 bin]# cp ../ software/redis-6.2.6/redis. conf ./ rainhey_ config

8. Modify rainhey_ In the configuration file under config, modify daemon yes

9. Start redis with the modified configuration file[ root@node1 bin]# redis-server rainhey_ config/redis. conf

10. Open the client to connect to the server, redis cli - P 6379

11. test
    

12. Check whether the Redis process is started
    

13. Turn off Redis service
    

Redis basic commands

Official order complete document

database

select 3    //select to switch databases. Redis has 16 databases (No. 0-15) by default, and the 0th database is used by default

dbsize      //Current database size

Set value

set name rainhey     //Key value pair
get name    //Value

keys *      //View all key s

empty

flushdb		//Empty the current database

flushall		//Empty all databases

Judge existence

exists name	     //Judge whether the key value exists; Returns 1 if it exists and 0 if it does not exist

move

move name 1   //Remove the key from the current database to database 1

Set expiration

expire age 20     //Set the expiration time (seconds) of the key, which will be automatically removed after expiration

ttl age       //View the remaining time of the key

type

type name      //View the type of key

Basic data type

String

Add

append name test     //Append the string "test" after the value of key name. If the key does not exist, it is equivalent to adding

length

strlen name      //Gets the string length of the value corresponding to the key

Addition and subtraction

incr view     //view value plus 1

decr view    //Value minus 1

incrby view 10      //view value plus 10

decrby view 11    //Value minus 11

Range value

getrange test 0 4       //Intercept the string with the subscript [0,4], and note that both sides are closed intervals; [0, - 1] represents the entire string

setrange test 1 ***     //Replacement string, 1 represents the subscript to start replacement, and * * * represents the value to start replacement

be overdue

setex key1 20 "hello"     //  setex( set with expire )

No settings exist

setnx key2 "hello"     // setnx (set if not exist)

Batch operation

mset k1 v1 k2 v2 k3 v3      // Batch setting value

mget k1 k2 k3      // Batch get value

msetnx k1 v1 k4 v4    // The key is set only when it does not exist, otherwise it is not set

object

set user:1 {name:zhangsan,age:3}      //Set the user:1 object and save its value with json string

mset user:1:name zhangsan user:1:age 55     //The key here is a clever design. user:{id}:{field}

Get settings

getset key2 v2        // Get the value before setting it

List

All List commands start with L
In addition, lpush is equivalent to entering the stack and rpush is equivalent to entering the queue. When reading, read the stack first and then the queue

lpush list one      //Add elements to the list

rpush list four   //Add element to list

Range value

lrange list 0 -1    //Take values from the list

Left and right removal

lpop list    //Remove the first element in the list

rpop list    //Remove the last element in the list

rpoplpush list list1    //Remove the last element lpush of the list into list1

Subscript value

lindex list 1    //Get the value of list subscript 1

List length

llen list    // Get list length

remove

lrem list 2 four    //Remove two values with the value four from the list

trim

ltrim list 0 3      //Only the value of list[0, 3] is reserved

replace

lset list 0 one      //Replaces the value of the specified subscript

interpolation

linsert list before world other   // Add an other before world

Set

Redis Set is an unordered Set of string type. Collection members are unique, which means that duplicate data cannot appear in the collection
Collections in Redis are implemented through hash tables, so the complexity of adding, deleting and searching is O(1)

sadd myset hello1   //add value

smembers myset      //View members

sismember myset hello1    //Determine whether hello1 is a member of myset

scard myset    //View the number of myset elements

srem myset hello1    //Remove hello1

srandmember myset 2       //Take out two at random

spop myset 1    //Removes a specified number of elements at random

smove myset1 myset2 one    //Move one from myset1 to myset2

sdiff set1 set2    //Difference set of set2 in set1

sinter set1 set2     //intersection

sunion set1 set2     //Union

Hash

Map set, key map. At this time, the value is the set of maps, which is not much different from string in essence

hset myhash field1 rainhey   //Save key value pair

hget myhash field1       //Value

hmset myhash field1 value1 field2 value2   //Batch save key value pair

hmget myhash field1 field2    //Batch value

hgetall myhash    //Get all key value pairs

hdel myhash field1   //Delete the key, and there is no corresponding value

hlen myhash     // View the logarithm of key value pairs

hkeys myhash      //Get the key of all key value pairs

hvals myhash     //Gets the value of all key value pairs

hincrby myhash field3 4       //Add 4 to the value corresponding to field3

hsetnx myhash field4 hello       //Creation does not exist and cannot be set

Hash is more suitable for object storage, and Sring is more suitable for string storage!

Zset (ordered set)

A value set K1 V1 is added on the basis of set; zset k1 score v1

zadd myzset 1 one       // Set a value
zadd myzset 3 three 4 four    // Set multiple values

zrange myzset 0 -1   // Value, sorted from small to large

zrangebyscore salary -inf +inf   // Sort by score within the specified range

zrangebyscore salary -inf +inf withscores   //Sort with score

zrem salary lisi    // Removes the specified element

zcard salary   // View the number of elements

zrevrange salary 0 -1     // Sort from small to large

zcount myset 1 3    // Gets the number of elements with a score between 1 and 3

Three special data types

Geospatial geographic location

There are only six commands

• The effective longitude is from - 180 degrees to 180 degrees.
• The effective latitude ranges from -85.05112878 degrees to 85.05112878 degrees

geoadd china:city 114.878872 30.459422 beijing    // Add geographic location, parameter key, longitude dimension name
geoadd china:city 116.413384 39.910925 guangzhou 79.920212 37.118336 hetian    //Many places

geopos china:city beijing     // Get the latitude and longitude of a place
geopos china:city shanghai beijing     // Many places

distance

• m is in meters
• km is expressed in kilometers
• mi is expressed in miles.
• ft is in feet

geodist china:city shanghai beijing    //Calculate distance
geodist china:city shanghai beijing km   // In km

georadius china:city 118 30 500 km     // Find cities with longitude and latitude 118 and 30 as the center and a radius of 500km
georadius china:city 119 30 200 km withdist      // Find with distance  
georadius china:city 119 30 200 km withcoord    //Find and bring the latitude and longitude coordinates
georadius china:city 119 30 200 km withcoord withdist count 2     //Find and bring the coordinate distance and limit the number of searches 
eoradiusbymember china:city city3 100 km     // Find other members whose members are center points

geohash china:city city1 city2 city3      //This command returns an 11 character geohash string, which converts two-dimensional latitude and longitude into one-dimensional string. The closer the string is, the closer the distance is

The underlying implementation principle of geo is Zset, so you can also use the Zset command to operate Geo
View all elements

Removing Elements

Hyperloglog cardinality statistics

Cardinality: the number of non repeating elements in the dataset

pfadd myset a b c d e f g h i j     //add to
pfadd myset2 i j k c v z h
pfcount myset     //Count, do not repeat
pfmerge myset3 myset myset2    // Merge sets myset, myset2 to myset3

If fault tolerance is allowed, you can use Hyperloglog!

If fault tolerance is not allowed, use set or your own data type!

Bitmap

As long as it is in two states, it can be recorded with bitmap operation binary bits

setbit sign 2 0    //2 represents the second position and 0 represents the position status

getbit sign 6     //Gets the status of the sixth location

bitcount sign       //Count the number of sign s with status 1

Redis basic transaction operations

Redis transaction essence: a collection of commands. All commands of a transaction will be serialized and executed in sequence during transaction execution
Redis single command keeps atomicity, but transaction does not guarantee atomicity
Redis transactions do not have the concept of isolation level
All commands are not executed directly in the transaction. They will be executed only after the execution command is initiated

• Start transaction multi
• Order to join the team
• Execute transaction exec
  
  discard discards the transaction, and the commands in the queue will not be executed
  
  If a command in a transaction has a compilation exception, all commands will not be executed; If a command in a transaction has a runtime exception (such as 1 / 0), other commands can be executed normally

Redis implements optimistic lock

Pessimistic lock: I think there will be problems at any time and lock it no matter what I do
Optimistic lock: I don't think there will be any problem at any time and lock will not be added. When updating the data, judge whether someone has modified the data during this period

watch key monitors the specified data, which is equivalent to optimistic locking

Normal execution

Under abnormal circumstances, start a client simulated queue jumping thread
Thread 1: exec has not been executed

At this point, thread 2: changed the value

Thread 1 execution: the result is null and the transaction is not executed

If the transaction fails to execute, unlock unwatch to obtain the latest value, and then lock the transaction

Jedis

The Java connection tool officially recommended by Redis uses Java to operate Redis middleware

1. Import dependency

		<dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>4.1.1</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.72</version>
        </dependency>

1. Code example

public class Test {
    public static void main(String[] args) {

        JSONObject jsonObject = new JSONObject();
        jsonObject.put("hello", "world");
        jsonObject.put("name", "rainhey");

        String result = jsonObject.toJSONString();


        // 1.new jedis object
        Jedis jedis = new Jedis("192.168.0.100", 6379);
        jedis.auth("123456");
        // 2. All the commands of jedis are all the commands learned before
        Transaction multi = jedis.multi();
        try{
            multi.set("user1", result);
            multi.set("user2", result);
            multi.exec();
        }catch (Exception e){
            multi.discard();
            e.printStackTrace();
        }
        finally {
            System.out.println(jedis.get("user1"));
            System.out.println(jedis.get("user2"));
            jedis.close();
        }
    }
}

Spring boot integrates Redis

quick get start

Springboot 2. After X, the original Jedis is replaced by Lettuce

Jedis: direct connection and multi thread operation are not safe. If you want to avoid insecurity, use jedis pool connection pool! More like BIO mode

Lettuce: with netty, instances can be shared among multiple threads. There is no thread insecurity! Thread data can be reduced, more like NIO mode

1. Import dependency

		<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>

1. configure connections

spring.redis.host=192.168.0.100
spring.redis.port=6379
spring.redis.password=123456

Pay attention to some connection pool related configurations. When using, always use Lettuce connection pool

1. test

@SpringBootTest
class RedisSpringApplicationTests {

    @Autowired
    private RedisTemplate redisTemplate;

    @Test
    void contextLoads() {

        /*In addition to basic operations, common methods can be operated through redisTemplate, such as transaction and basic CRUD
        redisTemplate.
        opsForValue: Operation string, similar to string opsforlist: operation list opsforset() opsforzset()
        opsForHash()   opsForGeo()    opsForHyperLogLog()*/
        
        redisTemplate.opsForValue().set("mykey", "rainhey");
        System.out.println(redisTemplate.opsForValue().get("mykey"));

        /*Some commands operate through Redis's connection object
        * */
        /*RedisConnection connection = redisTemplate.getConnectionFactory().getConnection();
        connection.flushAll();
        connection.flushDb();*/
    }
}

When viewing data in Redis, it is found that the data is garbled, which is related to the serialization of storage objects. Looking at the source code, it can be seen that Redis adopts JDK serialization by default, and the data seen by Redis is garbled

serialize

Java provides a mechanism of object serialization. In this mechanism, an object can be represented as a byte sequence, which includes the data of the object, the information about the type of the object and the type of data stored in the object; After the serialized object is written to the file, it can be read from the file and deserialized, that is, the type information of the object, the data of the object, and the data type in the object can be used to create a new object in memory

public class User {

    private String name;
    private int age;

}

@Test
    void test() throws JsonProcessingException {
        User user = new User("rainhey", 3);
        //Serialize, convert to json string, and pass
        String s = new ObjectMapper().writeValueAsString(user);
        redisTemplate.opsForValue().set("user", s);
        System.out.println(redisTemplate.opsForValue().get("user"));
    }

public class User implements Serializable {

    private String name;
    private int age;

}

@Test
    void test() throws JsonProcessingException {
        User user = new User("rainhey", 3);
        //Serialize, convert to json string, and pass
        //String s = new ObjectMapper().writeValueAsString(user);
        redisTemplate.opsForValue().set("user", user);
        System.out.println(redisTemplate.opsForValue().get("user"));
    }

Customize redisTemplate

@Configuration
public class RedisConfig {

    //Learn from the source code and write your own redisTemplate to cover the underlying
    @Resource
    RedisConnectionFactory redisConnectionFactory;   // Error in method parameter injection. Use Resource injection

    @Bean
    public RedisTemplate<String, Object> redisTemplate() {
        // For development convenience, string is generally used
        RedisTemplate<String, Object> template = new RedisTemplate();
        template.setConnectionFactory(redisConnectionFactory);

        /*
         * Serialization settings
         * Call the static method of RedisSerializer to return the serializer
         */
        // The key of key and hash adopts String serialization
        template.setKeySerializer(RedisSerializer.string());
        template.setHashKeySerializer(RedisSerializer.string());
        // The value of value and hash adopts Jackson serialization
        template.setValueSerializer(RedisSerializer.json());
        template.setHashValueSerializer(RedisSerializer.json());
        template.afterPropertiesSet();

        return template;
    }

}

Customize Redis tool class

RedisTemplate needs to be called frequently opForxxx affects efficiency. In work, these common public API s are often extracted and encapsulated into a tool class, and then the tool class is directly used to indirectly operate Redis

Tool reference blog:

https://www.cnblogs.com/zeng1994/p/03303c805731afc9aa9c60dbbd32a323.html

https://www.cnblogs.com/zhzhlong/p/11434284.html

Redis.config

• The configuration file is case insensitive
• network

bind 127.0.0.1 -::1    //Bind IP
protected-mode no    //Whether to turn on protection mode
port 6379    //port

• currency

daemonize yes    //Execute as a daemon
pidfile /var/run/redis_6379.pid   //If it is executed in the background daemon mode, a PID file needs to be specified

# journal
# Specify the server verbosity level.
# This can be one of:
# debug (a lot of information, useful for development/testing)
# verbose (many rarely useful info, but not a mess like the debug level)
# Notice (moderately verbose, what you want in production probability) / / production environment
# warning (only very important / critical messages are logged)
loglevel notice   //log level
logfile ""    //Log output file

databases 16   //Number of databases
always-show-logo no   //Whether to display LOGO pagoda

• snapshot

Persistence: the number of operations performed within a specified time will be persisted to the file rdb .aof
Redis is an in memory database. If there is no persistence, the data will be lost due to power failure

# Save 3600 1 / / if at least one Key is modified within 3600 s, it will be persisted
# save 300 100
# save 60 10000

stop-writes-on-bgsave-error yes    //Persistence error. Do you want to continue working
rdbcompression yes   //Whether to compress rdb files requires some cpu resources
rdbchecksum yes   //Check and verify errors when saving rdb files
dir ./     //Directory where rdb files are saved

• REPLICATION replication

replicaof <masterip> <masterport>    //Configure the host ip and port, which is equivalent to the command slaveof
masterauth <master-password>    //Configure host password

• SECURITY security

requirepass 123456    //Set password

• CLIENTS

maxclients 10000   //Client upper limit
maxmemory <bytes>    //redis configured maximum memory capacity

maxmemory-policy noeviction    //Processing strategy after the memory reaches the upper limit
 Policy, setting method, such as config set maxmemory-policy volatile-lru
1,volatile-lru: Only for those with expiration time set key conduct LRU((default)
2,allkeys-lru :  delete lru Algorithmic key
3,volatile-random: Random deletion is about to expire key
4,allkeys-random: Random deletion
5,volatile-ttl :  Delete expiring
6,noeviction :  Never expire, return error

• APPEND ONLY MODE (AOF configuration)

appendonly no     //It is not enabled by default. rdb is used by default
appendfilename "appendonly.aof"    //The name of the persistent file

# appendfsync always / / every modification will be synchronized
appendfsync everysec    //Execute once per second, and one second of data may be lost
# appendfsync no / / out of sync
  

Persistence RDB

Write the data set snapshot in memory to disk within the specified time interval, and read the snapshot file directly to memory when it is restored
The file saved by rdb is dump rdb, which can be configured in the configuration file

RDB principle

1. Redis calls forks and has both parent and child processes
2. The subprocess writes the data to a temporary RDB file
3. When the child process finishes writing the new RDB file, Redis replaces the original RDB file with the new RDB file and deletes the old RDB file

Trigger mechanism

1. When the save rule is satisfied, the rdb policy will be triggered automatically
2. Executing the flush command will also trigger our rdb policy
3. When you exit redis, rdb files will also be generated automatically

After triggering the mechanism, an rdb file will be generated automatically

recovery

Just put the rdb file in the redis startup directory. When redis starts, it will automatically check dump rdb, recover the data in it

Advantages and disadvantages

advantage:

1. Suitable for large-scale data recovery
2. The requirements for data integrity are not high

Disadvantages:

1. The operation needs a certain time interval. If redis goes down unexpectedly, the last modified data will be gone
2. The fork process will occupy a certain content space

Persistence AOF

AOF saves appendonly AOF file
It is not enabled by default. You need to configure it in the configuration file
principle
Record all the executed commands (read operation commands are not recorded), and then execute all the commands in the file when recovering
repair
If there is an error in the AOF file, redis cannot start normally and needs to modify the AOF file. Redis provides a tool redis check AOF -- fix. After the AOF file is repaired normally, redis can start normally

redis-check-aof --fix appendonly.aof   //Repair aof file

Advantages and disadvantages

advantage

1. Each modification will be synchronized, and the integrity of the file will be better
2. Sync once every second, and you may lose one second of data
3. Never synchronized, most efficient

shortcoming

1. Compared with data files, aof is much larger than rdb, and the repair speed is slower than rdb!
2. aof also runs slower than rdb, so the default configuration of redis is rdb persistence

Redis publish and subscribe

command	describe
PSUBSCRIBE pattern [pattern..]	Subscribe to one or more channels that match the given pattern
PUNSUBSCRIBE pattern [pattern..]	Unsubscribe from one or more channels that match the given pattern
PUBSUB subcommand [argument[argument]]	View subscription and publishing system status
PUBLISH channel message	Publish messages to specified channels
SUBSCRIBE channel [channel..]	Subscribe to a given channel or channels
UNSUBSCRIBE channel [channel..]	Unsubscribe from one or more channels

1. Start a client subscription channel and automatically listen
   
2. Start a client to send messages
   
3. The subscriber receives the message
   

principle
Each Redis server process maintains a Redis that represents the status of the server H / redisserver structure, PubSub of structure_ The channels attribute is a dictionary, which is used to save the information of subscribed channels. The key of the dictionary is the channel being subscribed, and the value of the dictionary is a linked list, which stores all clients subscribing to this channel
When a client subscribes, it is linked to the tail of the linked list of the corresponding channel. Unsubscribing is to remove the client node from the linked list

application

1. Message subscription: official account subscription, micro-blog attention, etc.
2. Multiplayer online chat room

Redis cluster

Master-slave replication

Master slave replication refers to copying data from one Redis server to other Redis servers. The former is called the master node and the latter is called the slave node. Data replication is one-way and can only be replicated from the master node to the slave node

effect

1. Data redundancy: master-slave replication realizes the hot backup of data, which is a way of data redundancy other than persistence
2. Fault recovery: when the master node fails, the slave node can temporarily replace the master node to provide services, which is a way of service redundancy
3. Load balancing: on the basis of master-slave replication, with read-write separation, the master node performs write operations and the slave node performs read operations to share the load of the server; Especially in the scenario of more reads and less writes, multiple slave nodes share the load and improve the concurrency
4. High availability cornerstone: master-slave replication or the basis for sentinel and cluster implementation

Environment configuration

Configure only the slave database, not the master database, because Redis is the master database by default

View the information of current Redis master-slave replication

info replication

1. Copy redis Conf is named redis79 conf,redis80.conf,redis81.conf
   Modify profile

redis79.conf: 
logfile "6379.log"
dbfilename dump6379.rdb

redis80.conf: 
port 6380
pidfile /var/run/redis_6380.pid
logfile "6380.log"
dbfilename dump6380.rdb

redis81.conf: 
port 6381
pidfile /var/run/redis_6381.pid
logfile "6381.log"
dbfilename dump6381.rdb

1. Start three Redis services, build one master and two slaves, and only configure slaves

Slave 80:

slaveof 127.0.0.1 6379

Slave 81:

slaveof 127.0.0.1 6379

View host replication

The real master-slave configuration is configured in the configuration file. We use commands here, which is temporary

Configure master-slave configuration in configuration file

replicaof <masterip> <masterport>    //Configure the host ip and port, which is equivalent to the command slaveof
masterauth <master-password>    //Configure host password

details

1. The slave can only read and cannot write. The host can read and write, but it is mostly used for writing. All data and information in the host will be actively saved by the slave

2. When the host is disconnected, the role of the slave will not change by default. The cluster only loses the write operation. When the host is restored, the slave will be restored to its original state

3. When the slave is disconnected, if the master-slave configured with the command line is used instead of the slave configured with the configuration file, the original slave will automatically recover to the host after restart. It is impossible to obtain the data of the host connected as the slave. If it is reconfigured as the slave of the previously connected host at this time, all the data of the host can be obtained
   As long as the host is reconnected, a full copy will not be executed automatically; After that, the newly added data on the host is incrementally copied to the slave

4. Link model
   
   Similarly, only the primary node can write

5. Slave node to master node command

SLAVEOF no one

Sentinel mode

When the primary server goes down, it is necessary to manually switch a slave server to the primary server, which requires manual intervention, which is laborious and laborious, and the service will not be available for a period of time

Sentinel mode: automatically monitor whether the host fails. When the host fails, the slave will be automatically switched to the host according to the number of votes. Sentinel is an independent process, which monitors multiple running Redis instances by sending commands to the Redis server and waiting for responses

There may be problems when one sentinel monitors the Redis server. We can use multiple sentinels for monitoring, and each sentinel will also monitor, thus forming a multi sentinel mode

Assuming that the main server is down and sentinel 1 detects this result first, the system will not immediately carry out the failover process. Only sentinel 1 subjectively thinks that the main server is unavailable. This phenomenon is called subjective offline. When the subsequent sentinels also detect that the main server is unavailable and the number reaches a certain value, a vote will be held between sentinels, and the voting result will be initiated by one sentinel, The failover operation is carried out. Through the subscription and publishing mode, each sentinel is allowed to switch the monitored from the server to the host. This process is called objective offline

test

First build a master-slave mode

1. Write sentry configuration

vim sentinel.conf

sentinel monitor myredis 127.0.0.1 6379 1   
//  myredis uses its own monitoring name
//  Monitoring ip and ports
//  1 means that when a sentinel subjectively thinks that the host is disconnected, he can objectively think that the host is faulty, and then start electing a new host

1. Open the sentry

redis-sentinel ./rainhey_config/sentinel.conf

3. Turn off the host 6379 at this time
4. By observing 6380 and 6381, it can be found that 81 has automatically become the master and 80 has become its slave

5. Sentry log

After the host is disconnected, a server will be selected from the slave as the host (voting algorithm)
If you connect back to the previous host at this time, you can find that the previous host has become a slave

Advantages and disadvantages

advantage:

1. Sentinel cluster is based on master-slave replication mode. It has all the advantages of master-slave replication
2. The master and slave can be switched, the fault can be transferred, and the availability of the system is better
3. Sentinel mode is the upgrade of master-slave mode. It is more robust from manual to automatic

Disadvantages:

1. Redis is not easy to expand online. Once the cluster capacity reaches the upper limit, online expansion will be very troublesome
2. The configuration of sentinel mode is actually very troublesome. There are many configuration items

Full configuration of sentinel mode

# Example sentinel.conf
 
# The port on which the sentinel sentinel instance runs is 26379 by default
port 26379
 
# sentinel's working directory
dir /tmp
 
# The ip port of the redis master node monitored by sentry sentinel 
# The master name can be named by itself. The name of the master node can only be composed of letters A-z, numbers 0-9 and the three characters ". -" form.
# Quorumwhen the sentinel of these quorum s thinks that the master master node is lost, it objectively thinks that the master node is lost
# sentinel monitor <master-name> <ip> <redis-port> <quorum>
sentinel monitor mymaster 127.0.0.1 6379 1
 
# When the requirepass foobared authorization password is enabled in the Redis instance, all clients connecting to the Redis instance must provide the password
# Set the password of sentinel sentinel connecting master and slave. Note that the same authentication password must be set for master and slave
# sentinel auth-pass <master-name> <password>
sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
 
 
# After the specified number of milliseconds, the master node does not respond to the sentinel sentinel. At this time, the sentinel subjectively thinks that the master node goes offline for 30 seconds by default
# sentinel down-after-milliseconds <master-name> <milliseconds>
sentinel down-after-milliseconds mymaster 30000
 
# This configuration item specifies the maximum number of slave s that can synchronize the new master at the same time when a failover active / standby switch occurs,
The smaller the number, the better failover The longer it takes,
But if this number is bigger, it means more slave because replication Not available.
You can ensure that there is only one at a time by setting this value to 1 slave Is in a state where the command request cannot be processed.
# sentinel parallel-syncs <master-name> <numslaves>
sentinel parallel-syncs mymaster 1
 
 
 
# Failover timeout can be used in the following aspects: 
#1. The interval between two failover s of the same sentinel to the same master.
#2. When a slave synchronizes data from a wrong master, the time is calculated. Until the slave is corrected to synchronize data to the correct master.
#3. The time required to cancel an ongoing failover.  
#4. During failover, configure the maximum time required for all slaves to point to the new master. However, even after this timeout, the slave will still be correctly configured to point to the master, but it will not follow the rules configured by parallel syncs
# The default is three minutes
# sentinel failover-timeout <master-name> <milliseconds>
sentinel failover-timeout mymaster 180000
 
# SCRIPTS EXECUTION
 
#Configure the script to be executed when an event occurs. You can notify the administrator through the script. For example, send an email to notify relevant personnel when the system is not running normally.
#There are the following rules for the running results of scripts:
#If the script returns 1 after execution, the script will be executed again later. The number of repetitions is currently 10 by default
#If the script returns 2 after execution, or a return value higher than 2, the script will not be executed repeatedly.
#If the script is terminated due to receiving the system interrupt signal during execution, the behavior is the same as when the return value is 1.
#The maximum execution time of a script is 60s. If it exceeds this time, the script will be terminated by a SIGKILL signal and then re executed.
 
#Notification script: this script will be called when any warning level event occurs in sentinel (such as subjective failure and objective failure of redis instance),
#At this time, the script should notify the system administrator about the abnormal operation of the system through e-mail, SMS, etc. When calling the script, two parameters will be passed to the script,
#One is the type of event,
#One is the description of the event.
#If sentinel If the script path is configured in the conf configuration file, you must ensure that the script exists in this path and is executable, otherwise sentinel cannot be started successfully.
#Notification script
# sentinel notification-script <master-name> <script-path>
  sentinel notification-script mymaster /var/redis/notify.sh
 
# Client reconfiguration master node parameter script
# When a master changes due to failover, this script will be called to notify the relevant clients of the change of the master address.
# The following parameters will be passed to the script when calling the script:
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
# At present, < state > is always "failover",
# < role > is one of "leader" or "observer". 
# The parameters from IP, from port, to IP and to port are used to communicate with the old master and the new master (i.e. the old slave)
# This script should be generic and can be called multiple times, not targeted.
# sentinel client-reconfig-script <master-name> <script-path>
sentinel client-reconfig-script mymaster /var/redis/reconfig.sh

Cache penetration and avalanche

Cache penetration

concept
When the user wants to query some data, there is no in Redis memory database, that is, the cache misses, so he initiates a request to the persistence layer database; When there are many users, Redis cache misses and requests the persistence layer database, which will cause great pressure on the persistence layer database. This is cache penetration

Solution

1. Bloom filter

2. Cache empty objects

If a request is not found in the cache and database, an empty object of the request is placed in the cache to process subsequent requests

Cache breakdown (too large amount, data expiration)

Cache breakdown refers to the existence of a hot key in the cache, which is accessed intensively by large concurrency. When the key fails, the continuous large concurrency directly requests the persistence layer database, resulting in excessive pressure on the database

Solution

1. Hotspot data never expires
2. Add mutex lock (distributed lock)
   Using distributed locks ensures that each key has only one thread to query the back-end service at the same time. Other threads do not obtain distributed locks and need to wait. This shifts the pressure of high concurrency to distributed locks

Cache avalanche

In a certain period of time, the cache set expires or Redis goes down

Solution

1. Redis high availability: build redis cluster
2. Current limiting degradation: after the cache fails, the number of threads reading and writing to the database cache is controlled by locking or queuing. For example, for a key, only one thread is allowed to query data and write cache, while other threads wait
3. Data preheating: access the possible data in advance, so that part of the data that may be accessed in large quantities will be loaded into the cache; Before a large concurrent access is about to occur, manually trigger the loading of different key s in the cache and set different expiration times to make the time point of cache invalidation as uniform as possible