1, Introduction to Redis
NoSQL (Not Only SQL), which means Not Only SQL, but also non relational database.
with the development of internet web2 With the rise of web2.0 website, the traditional relational database is dealing with web2.0 0 websites, especially the super large-scale and highly concurrent SNS type web2 0 pure dynamic website has been unable to meet its needs, exposing many insurmountable problems, while non relational database has developed very rapidly due to its own characteristics. The generation of NoSQL database is to solve the challenges brought by large-scale data collection and multiple data types, especially the problems of big data application, including the storage of large-scale data. (for example, Google or Facebook collects trillions of bits of data for their users every day). These types of data storage do not need a fixed mode and can be expanded horizontally without redundant operations.
NoSQL representatives: MongDB, Redis, Memcache
Redis is an open source log and key value database written in ANSIC language, supporting network, memory based and persistent, and provides API s in multiple languages. Since March 15, 2010, the development of Redis has been hosted by VMware.
Redis is a key value storage system. Five data types are supported: string, hash, list, set and zset. These data types support push/pop, add/remove, intersection, union, difference and richer operations, and these operations are atomic. On this basis, redis supports various sorting methods. Like memcached, data is cached in memory to ensure efficiency. The difference is that redis will periodically write the updated data to the disk or write the modification operation to the additional record file, and on this basis, it realizes master-slave synchronization.
redis is a high-performance key value database. The emergence of redis largely compensates for the shortage of keyvalue storage such as memcached, and can play a good supplementary role to relational databases on some occasions. It provides Java, Python, Ruby, Erlang and PHP clients, which is very convenient to use.
all data of redis is saved in memory and then saved to disk asynchronously from time to time (this is called "semi persistent mode"); You can also write every data change to an append only file (AOF) (this is called "full persistence mode"). Since the data of redis is stored in memory, if persistence is not configured, all the data will be lost after redis is restarted. Therefore, it is necessary to turn on the persistence function of redis to save the data to the disk. After redis is restarted, the data can be recovered from the disk. Redis provides two methods for persistence. One is RDB persistence (the principle is to periodically dump the database records of Reids in memory to the RDB persistence on disk), and the other is AOF (append only file) persistence (the principle is to write the operation logs of Reids to the file in the form of append).
2, Install redis service
1. Use the Xiaopi system to install and start the redis service.
You can use RedisClient or redis cli Exe to view the data in the database.
Common Redis commands:
ping Is the test database connected select 6 Select the 6th database set a 15 with a Save a value of 15 for the key get a obtain a Value of keys * View all keys in the library flushdb Empty the current database flushall Empty all databases
Create a new Java project based on maven. And import the redis connection dependency.
<dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> <version>3.7.0</version> </dependency>
Create a Java class and add the main method test
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
System.out.println(jedis.ping());
}
When the return value is PONG, the database connection is successful. For faster performance, connections are generally obtained through the connection pool.
3, Use connection pool
The following connection pool uses the hungry man mode in the singleton mode. The relevant configuration is limited to jedis-2.1.0, and the configuration of other versions is different.
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;
public class JedisPoolUtil {
private static volatile JedisPool jedisPool = null;
private JedisPoolUtil() {
}
public static Jedis getJedisInstance() {
if (null == jedisPool) {
synchronized (JedisPoolUtil.class) {
JedisPoolConfig poolConfig = new JedisPoolConfig();
//The maximum number of idle connections is 8 by default
poolConfig.setMaxIdle(32);
//Gets the maximum number of milliseconds to wait for a connection. If it times out, throw an exception. Less than zero: the blocking time is uncertain. The default is - 1
poolConfig.setMaxWaitMillis(100 * 1000);
//Check the validity when obtaining the connection. The default is false
poolConfig.setTestOnBorrow(true);
jedisPool = new JedisPool(poolConfig, "127.0.0.1", 6379);
}
}
return jedisPool.getResource();
}
public static void release(Jedis jedis) {
if (null != jedis) {
jedisPool.returnResource(jedis);
}
}
}
Get redis connection from connection pool when using
Jedis jedis = JedisPoolUtil.getJedisInstance(); ..... JedisPoolUtil.release(jedis);
4, String type operation
redis is a key value storage system. Five data types are supported: string, hash, list, set and zset.
The string in Redis is a sequence of bytes. Strings in Redis are binary safe, which means that their length is not determined by any special termination character. Therefore, anything up to 512 megabytes can be stored in a string.
public static void main(String[] args) {
Jedis jedis = new Jedis( "127.0.0.1", 6379 );
jedis.select( 0 );//Select the database for the current operation
System.out.println( jedis.keys( "*" ) );//View all keys in the current database
jedis.set( "hello", "helloword" );//Store data
jedis.append( "hello", " haiwen" );//Add
System.out.println( jedis.get( "hello" ) );//Fetch data
jedis.del( "user" );//Delete key
jedis.mset( "bb","bb1","cc","cc1");
//Store multiple key value pairs at one time (it must be an even number. If it is an odd number, it will throw ERR wrong number of arguments for MSET)
//jedis.incr( "hello" );
//jedis.incrBy( "hello",50 );
//Add 1 to the value of aa (the modified object must be a number. If it is a character, throw ERR value is not an integer or out of range)
//jedis.decr( "aa" );// Subtract the value of AA by 1
jedis.setex( "ff",12,"From the president of the Republic" );//Set the survival time of the key value pair to 12 seconds
jedis.flushDB();//Delete all data in the current database
jedis.flushAll();//Delete all data in the database
jedis.disconnect();
}
5, List type operation
Redis list is just a string list, sorted by insertion order. You can add elements to the head or tail of redis list.
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
//jedis.lpush( "list", "Java" );
//jedis.lpush( "list", "UI" );
//jedis. Lpush ("list", "graphic design")// Press an object into the top of the list
//String list = jedis.lpop( "list" );// Take an object from the top, and there will be one less thing in it after taking it
//The first is the key, the second is the start position, and the third is the end position, jedis Len get length - 1 means get all
List<String> list = jedis.lrange("list", 0, -1);//Check the values in the list
for (String s : list) {
System.out.println(s);
}
System.out.println(jedis.llen("list"));//Get length
// jedis. Rpush ("list", "last")// Insert data at the bottom of the stack
//jedis.lset( "list" ,2,"aaa" );// Set the value with index 2 under the list key (overwrite if it exists)
jedis.lrem("list", 2, "aaa");
//Delete key value pairs with list key and aaa value. The number of deleted pairs is 2. If the number of deleted pairs is 0, all pairs will be deleted. After deletion, the indexes inside are sorted automatically
jedis.disconnect();
}
6, Hash type operation
Redis Hashes are a collection of key value pairs. Redis hash / hash is the mapping between string fields and string values. Therefore, you can think of Hashes as objects.
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
Map<String, String> map = new HashMap<>();
map.put("name", "xinxin");
map.put("age", "22");
map.put("qq", "123456");
jedis.hmset("user", map);
//The delete key is the subkey in user, and there can be multiple subkeys
//jedis.hdel( "user","qq","..." );
//The first parameter is the key of the whole map, and the latter parameter is the key in the map. You can write whatever value you need,
// If the corresponding attribute is not found, the position is empty
List<String> hmget = jedis.hmget("user", "qq", "age", "name", "aa");
System.out.println(hmget);
System.out.println(jedis.hlen("user")); // Returns the number of values stored in the key whose key is user. 2
System.out.println(jedis.exists("user"));// Whether there is a record whose key is user returns true
System.out.println(jedis.hkeys("user"));// Returns all key s in the map object
System.out.println(jedis.hvals("user"));// Returns all value s in the map object
jedis.disconnect();
}
7, Collection type operation
Redis collection is an unordered collection of strings. In redis, you can add, delete and test the time O(1) complexity of members.
public static void main(String[] args) {
Jedis jedis = new Jedis( "127.0.0.1", 6379 );
//jedis.flushDB();
jedis.sadd( "user", "liuling" );
jedis.sadd( "user", "xinxin" );
jedis.sadd( "user", "ling" );
jedis.sadd( "user", "zhangxinxin" );
jedis.sadd( "user", "who" );
jedis.sadd( "user", "who" );//Duplicate is not allowed, and sorting
Set<String> user = jedis.smembers( "user" );
for (String s : user) {
System.out.println( s );
}
jedis.srem( "user", "who" );// remove
System.out.println( jedis.sismember( "user", "who" ) );// Determine whether who is an element of the user collection
System.out.println( jedis.scard( "user" ) );// Returns the number of elements in the collection
jedis.disconnect();
}
8, Sortable set operation
Redis sortable sets are similar to redis sets and are non duplicate character sets. The difference is that each member of the sort set is associated with a score, which is used to sort the sort set from the minimum score to the maximum score. Although the member is unique, the score value can be repeated.
public static void main(String[] args) {
Jedis jedis = new Jedis( "127.0.0.1", 6379 );
jedis.zadd( "hackers", 1940, "Alan Kay" );
jedis.zadd( "hackers", 1953, "Richard Stallman" );
jedis.zadd( "hackers", 1965, "Yukihiro Matsumoto" );
jedis.zadd( "hackers", 1916, "Claude Shannon" );
jedis.zadd( "hackers", 1969, "Linus Torvalds" );
jedis.zadd( "hackers", 1912, "Alan Turing" );
Set<String> setValues = jedis.zrange( "hackers", 0, -1 );
System.out.println( setValues );
Set<String> setValues2 = jedis.zrevrange( "hackers", 0, -1 );
System.out.println( setValues2 );
// wipe data
System.out.println( jedis.flushDB() );
// Add data
jedis.zadd( "zset", 10.1, "hello" );
jedis.zadd( "zset", 10.0, ":" );
jedis.zadd( "zset", 9.0, "zset*" );
jedis.zadd( "zset", 11.0, "zset!" );
// Number of elements
System.out.println( jedis.zcard( "zset" ) );
// Element subscript
System.out.println( jedis.zscore( "zset", "zset*" ) );
// Set subset
System.out.println( jedis.zrange( "zset", 0, -1 ) );
// Delete element
System.out.println( jedis.zrem( "zset", "zset!" ) );
System.out.println( jedis.zcount( "zset", 9.5, 10.5 ) );
// Entire set value
System.out.println( jedis.zrange( "zset", 0, -1 ) );
jedis.disconnect();
}
9, List sorting
In development, we often use Jedis to sort the list.
public static void main(String[] args) {
Jedis jedis = new Jedis( "127.0.0.1", 6379 );
jedis.rpush("a", "1");
jedis.lpush("a","6");
jedis.lpush("a","3");
jedis.lpush("a","9");
System.out.println(jedis.lrange("a",0,-1));// [9, 3, 6, 1]
System.out.println(jedis.sort("a")); //[1, 3, 6, 9] / / enter the sorted result
System.out.println(jedis.lrange("a",0,-1));//[9, 3, 6, 1]
jedis.disconnect();
}
10, Transactions and locks
Multiple redis operations can also support transactions.
public static void main(String[] args) {
Jedis jedis = new Jedis( "127.0.0.1", 6379 );
jedis.flushDB();
Transaction transaction = jedis.multi();
try {
transaction.set("aa", "aa12");
int i= 1/0;
transaction.set("bb", "bb12");
transaction.exec();//Submit
} catch (Exception e) {
transaction.discard();//RollBACK
}
jedis.disconnect();
}
redis supports optimistic locking mechanism by default
public boolean transMethod() throws InterruptedException {
Jedis jedis = new Jedis("127.0.0.1", 6379);
int balance;// Available balance
int debt;// bill
int amtToSubtract = 10;// Actual amount
jedis.watch("balance");
// jedis.set("balance","5");// This sentence should not appear. It is convenient to give a lecture. Simulate that other programs have modified this entry
Thread.sleep(10000);
balance = Integer.parseInt(jedis.get("balance"));
if (balance < amtToSubtract) {
jedis.unwatch();
System.out.println("Sorry, your credit is running low");
return false;
} else {
System.out.println("***********transaction");
Transaction transaction = jedis.multi();
transaction.decrBy("balance", amtToSubtract);
transaction.incrBy("debt", amtToSubtract);
if (transaction.exec() == null) {
System.out.println("Modified midway");
} else {
System.out.println("Successful execution");
}
balance = Integer.parseInt(jedis.get("balance"));
debt = Integer.parseInt(jedis.get("debt"));
System.out.println("*******" + balance);
System.out.println("*******" + debt);
return true;
}
}
/**
* Generally speaking, the watch command is to mark a key. If a key is marked, if the key is modified by others before committing the transaction, the transaction will fail. In this case, you can usually try again in the program.
* First, mark the key balance, and then check whether the balance is sufficient. If it is insufficient, cancel the mark and make no deduction; If enough, start the transaction for update operation,
* If the key balance is modified by others during this period, it will not be modified when exec is executed. Usually, such errors can be caught in the program and executed again until it succeeds.
*/
public static void main(String[] args) throws InterruptedException {
TestWatch test = new TestWatch();
boolean retValue = test.transMethod();
System.out.println("main retValue-------: " + retValue);
}
11, Integration with spring
Add the host and port that Redis depends on and configures jedis
<bean id="jedis" class="redis.clients.jedis.Jedis">
<constructor-arg name="host" value="127.0.0.1"></constructor-arg>
<constructor-arg name="port" value="6379"></constructor-arg>
</bean>
Get Jedis from the spring container or inject it directly
public static void main(String[] args) {
ApplicationContext applicationContext = new ClassPathXmlApplicationContext( "spring-context.xml" );
Jedis bean = applicationContext.getBean(Jedis.class);
System.out.println(bean.ping());
}
12, Cache avalanche
For system A, assuming 5000 requests per second in the peak period every day, the cache could have carried 4000 requests per second in the peak period, but the cache machine unexpectedly went down completely. The cache hangs. At this time, all 5000 requests in one second fall into the database. The database will not be able to carry them. It will call the police and then hang up. At this time, if no special scheme is adopted to deal with the fault, the DBA is very anxious and restarts the database, but the database is immediately killed by new traffic.
This is cache avalanche.
About three years ago, a well-known Internet company in China had an avalanche due to a cache accident, and all the background systems collapsed. The accident lasted from that afternoon to 3 ~ 4 a.m. in the evening, and the company lost tens of millions. The solutions of cache avalanche are as follows.
In advance: redis is highly available, master-slave + sentinel, and redis cluster to avoid total collapse.
In fact: local ehcache cache + hystrix current limiting & degradation to avoid MySQL being killed.
Afterwards: redis is persistent. Once restarted, it will automatically load data from the disk and quickly recover cached data.
The user sends A request. After receiving the request, system A first checks the local ehcache cache, and then checks redis if it is not found. If neither ehcache nor redis exists, check the database again and write the results in the database into ehcache and redis.
The current limiting component can set the requests per second, how many can pass the component, and what about the remaining failed requests? Let's go! You can return some default values, or friendly tips, or blank values.
Benefits:
The database will never die. The current limiting component ensures that only how many requests can pass per second.
As long as the database does not die, that is, for users, 2 / 5 of the requests can be processed.
As long as 2 / 5 of the requests can be processed, it means that your system is not dead. For users, it may be that they can't swipe the page after clicking several times, but they can swipe it once more.
Cache penetration
For system A, assume 5000 requests per second, and 4000 of them are malicious attacks by hackers.
The 4000 attacks sent by hackers can't be found in the cache. Every time you go to the database, you can't find them.
Take a chestnut. The database id starts from 1. As a result, all the request IDS sent by hackers are negative numbers. In this way, there will be no requests in the cache. Each time, the request will "treat the cache as nothing" and query the database directly. The cache penetration of this malicious attack scenario will directly kill the database.
The solution is very simple. Every time system A does not find it in the database, it writes A null value to the cache, such as set -999 UNKNOWN. Then set an expiration time, so that the next time the same key is accessed, you can directly get data from the cache before the cache expires.
Buffer breakdown
Cache breakdown means that a key is very hot and frequently accessed. It is in a centralized and highly concurrent access situation. When the key fails, a large number of requests break through the cache and directly request the database, which is like cutting a hole in a barrier.
The solution is also very simple. You can set the hotspot data to never expire; Or implement the mutex lock based on redis or zookeeper, wait for the first request to build the cache, and then release the lock, so that other requests can access data through the key.
13, Install redis cluster (from)
Redis cluster management tool redis trib RB relies on the ruby environment
First, you need to install the ruby environment: install ruby
yum install ruby yum install rubygems
Install ruby and redis interface programs
Copy redis-3.0.0 Gem to / usr/local:
gem install /usr/local/redis-3.0.0.gem
Create the redis cluster directory under / usr/local, and create 7001 and 7002 under it.. 7006 catalog
6 copies of the installed redis
Copy the files in the redis installation directory bin to each 700X directory, and redis trib. In the redis source code directory src Copy RB to the redis cluster directory.
Modify each 700 X Directory redis.conf Profile: port 700X #bind 192.168.101.3 cluster-enabled yes
Start each redis
Enter 7001, 7002,... 7006 directories respectively and execute:
./redis-server ./redis.conf
After all nodes are started, check the startup of all nodes
ps aux|grep redis root 1281 0.0 0.9 140888 9544 ? Ssl 20:26 0:00 ./redis-server *:7001 [cluster] root 1288 0.0 0.9 140888 9228 ? Ssl 20:26 0:00 ./redis-server *:7002 [cluster] root 1295 0.0 0.9 140888 9548 ? Ssl 20:26 0:00 ./redis-server *:7003 [cluster] root 1302 0.0 0.9 140888 9536 ? Ssl 20:26 0:00 ./redis-server *:7004 [cluster] root 1309 0.0 0.9 140888 9528 ? Ssl 20:26 0:00 ./redis-server *:7005 [cluster] root 1316 0.0 0.9 140888 9380 ? Ssl 20:26 0:00 ./redis-server *:7006 [cluster]
Execute cluster command
./redis-trib.rb create --replicas 1 192.168.131.102:7001 192.168.131.102:7002 192.168.131.102:7003 192.168.131.102:7004 192.168.131.102:7005 192.168.131.102:7006
If you execute the cluster command for the second time, you need to delete the previously generated data
cd 7001/bin/ rm -rf *.aof rm -rf *.rdb rm -rf nodes.conf ./redis-server ./redis.conf cd .. cd .. cd 7002/bin/ rm -rf *.aof rm -rf *.rdb rm -rf nodes.conf ./redis-server ./redis.conf cd .. cd .. cd 7003/bin/ rm -rf *.aof rm -rf *.rdb rm -rf nodes.conf ./redis-server ./redis.conf cd .. cd .. cd 7004/bin/ rm -rf *.aof rm -rf *.rdb rm -rf nodes.conf ./redis-server ./redis.conf cd .. cd .. cd 7005/bin/ rm -rf *.aof rm -rf *.rdb rm -rf nodes.conf ./redis-server ./redis.conf cd .. cd .. cd 7006/bin/ rm -rf *.aof rm -rf *.rdb rm -rf nodes.conf ./redis-server ./redis.conf cd .. cd .. ./redis-trib.rb create --replicas 1 192.168.159.130:7001 192.168.159.130:7002 192.168.159.130:7003 192.168.159.130:7004 192.168.159.130:7005 192.168.159.130:7006
When you see
M: 9ead02ced1e47fc92b2d43f39eadaaeebe110664 192.168.159.130:7001 slots:0-5460 (5461 slots) master M: c4590e1a1428a363545a29b9910b880791630416 192.168.159.130:7002 slots:5461-10922 (5462 slots) master M: fd4abd8efab2864cd1e0066843e10f96161dc289 192.168.159.130:7003 slots:10923-16383 (5461 slots) master S: b3ec510d2859203fb2ceabba35fc3d38207847a6 192.168.159.130:7004 replicates 9ead02ced1e47fc92b2d43f39eadaaeebe110664 S: a195de8609128e4bcd0d2c5bc236cf79935748f3 192.168.159.130:7005 replicates c4590e1a1428a363545a29b9910b880791630416 S: fb8c46bca7e4d9b13a75dfa81f1448953087b6f0 192.168.159.130:7006 replicates fd4abd8efab2864cd1e0066843e10f96161dc289 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join.. >>> Performing Cluster Check (using node 192.168.159.130:7001) M: 9ead02ced1e47fc92b2d43f39eadaaeebe110664 192.168.159.130:7001 slots:0-5460 (5461 slots) master M: c4590e1a1428a363545a29b9910b880791630416 192.168.159.130:7002 slots:5461-10922 (5462 slots) master M: fd4abd8efab2864cd1e0066843e10f96161dc289 192.168.159.130:7003 slots:10923-16383 (5461 slots) master M: b3ec510d2859203fb2ceabba35fc3d38207847a6 192.168.159.130:7004 slots: (0 slots) master replicates 9ead02ced1e47fc92b2d43f39eadaaeebe110664 M: a195de8609128e4bcd0d2c5bc236cf79935748f3 192.168.159.130:7005 slots: (0 slots) master replicates c4590e1a1428a363545a29b9910b880791630416 M: fb8c46bca7e4d9b13a75dfa81f1448953087b6f0 192.168.159.130:7006 slots: (0 slots) master replicates fd4abd8efab2864cd1e0066843e10f96161dc289 [OK] All nodes agree about slots configuration.
Indicates that the cluster is started successfully
redis cluster clear cache
redis-cli -c -h 59.215.230.36 -p 3433 -a redis_2018
redis_2018 database password (all passwords on the cluster need to be cleared)
14, redis master-slave replication
Due to the short reading interval, unreadability may occur for the first time
public static void main(String[] args) {
Jedis jedis_M = new Jedis("127.0.0.1",6379);
Jedis jedis_S = new Jedis("127.0.0.1",6380);
jedis_S.slaveof("127.0.0.1",6379);
jedis_M.set("class","1122V2");
String result = jedis_S.get("class");
System.out.println(result);
}