Phase II NOSQLday01

I Redis (in memory nosql database)

  1. NoSQL and RDBMS
    1.1: the characteristics of RDBMS (relational database mysql): it reflects the relationship between data, supports transactions, ensures the integrity and stability of business, and has good performance with a small amount of data. However, high concurrency will lead to database crash.
    1.2 NoSQL (non relational database Redis, HBASE, MongoDB) features: it is generally used for data caching or database storage in high concurrency and high performance scenarios. It has fast reading and writing speed and high concurrency. It is not as stable as RDBMS and is not friendly to transactional support.
    Reading pays attention to concurrency and writing pays attention to security.
  2. Redis functions and application scenarios:
    2.1 definition: the memory based distributed nosql database stores all data in memory and has a persistence mechanism. Every time redis restarts, the data will be reloaded from the file to memory, and all reads and writes are only based on memory
    2.2 function: high performance and high concurrency data storage, reading and writing
    2.3 features: good interaction with hardware, fast data reading and writing based on memory, distributed (expanded and stable), kv structured database, with various rich data types (String, hash, list, set, zset)
    2.4 application scenarios: cache (high concurrency, large amount of data cache, temporary cache), database (high performance, small amount of data read and write, big data storage, real-time processing results), message middleware (message queue, big data generally do not use kafka)
  3. Stand alone deployment of redis:
    3.1 upload and decompress, and install the C language compiler:
yum -y install gcc-c++ tcl

3.2 compilation and installation
Enter the source directory: / export/server/redis-3.2.8 / compile make; Installation: make PREFIX=/export/server/redis-3.2.8-bin install
3.3 modify the configuration (copy the configuration file):

 cp /export/server/redis-3.2.8/redis.conf  /export/server/redis-3.2.8-bin/

Create log file

 mkdir -p /export/server/redis-3.2.8-bin/logs

Create data file

 mkdir -p /export/server/redis-3.2.8-bin/datas

Modify configuration:
vim redis.conf
Change the bound machine address:

Background operation

Log storage location

Persistent data storage location in memory

Create a soft connection to redis-3.2.8-bin: ln -s redis-3.2.8-bin redis
Modify profile

Startup: edit the startup script in the bin directory of redis:


${REDIS_HOME}/bin/redis-server ${REDIS_HOME}/redis.conf

Add permission: CHMOD U + X / export / server / redis / bin / redis start sh

Port: 6379
Client connection: redis cli - H node1 - P 6379
shutdown closes the server. Or: kill -9 `cat /var/run/ ``

  1. Data structure and data type of redis
    Data structure: kv structure exists k as unique identifier and is fixed as string type; v: The real stored data can be string, hash, list, set, zset and bitmap. hypeloglog.
    4.1 string is used to store a single string and store the result
    pv: page views. Every time a user visits a page, pv will be increased by one
    uv: user visits. Each user is counted as one uv
    4.2 hash: map set, which is used to store the values of multiple attributes
    4.3 list: list, orderly and repeatable, can store multiple values
    4.4 set: set removes duplicate values and is unique
    It combines the features of map, top, and tre: list, which are not ordered

  2. General commands of redis:
    5.1 keys: list all keys of the current database
    5.2 del key: delete key
    5.3 exists key: determine whether the key exists
    5.4 type key: view type
    5.5 expire key 20s: the key expires after 20s
    5.6 ttl key: how long does the key expire
    5.7 select n: switch the database. By default, there are 16 databases starting from 0. The number of databases can be modified through the configuration file db0
    5.8 move key n: move a key to a database
    5.9 flushdb: clear database
    5.10 flush: clear the key s of all databases

  3. Operation on string:
    6.1 set : set s1 hadoop

    6.2 get key: get the value of the specified key
    get s1

    6.3 set the value of the given key to value and return the old value of the key
    getset key value

    6.4 set multiple string type kv: mset k1 v1 k2 v2

    6.5 get multiple value s: mget k1 k2

    6.6 build preemptive lock and use it with expire. It can only be used to add data. When k does not exist, add setnx k v

    6.7 incr: used to increment the string of numeric type by 1, generally used for counter incr k

    6.8 specify a fixed step for the growth of numeric strings: incrby k n

    6.9 decrement of numerical data by 1: decr k

    6.10 decrby: decrease by K n according to the specified step size

    6.11 strlen: counts the length of the string strlen k

    Getrange: used to intercept string 12.6

  4. hash type
    7.1 hset: used to add an attribute hset k V for a key

    7.2hmset: assign value to a k in batch:
    hmset K k1 v1 k2 v2 ......

    7.2 hget: used to get a property value of a k

    7.3hmget: get the values of multiple attributes of a k in batch
    hmget k k1 k2 k3 k4

    7.4hgetall: get the value of all attributes hgetall k

    7.5hdel: delete an attribute
    hdel k k1 k2 k3

    7.6 hlen: count the total number of attributes of value corresponding to k
    hlen k

    7.7 hexists: judge whether the k contains this attribute
    hexists k k

    7.8 hvals: get the value of all attributes

    7.9 hkeys: get all attributes

  5. Common commands of type list
    8.1 lpush: put each element to the left of the set and put it in the left order
    lpush k e1 e2
    8.2 rpush: put each element on the right side of the set and put it in the right order
    rpush k e1 e2
    8.3lrange: obtain the data of elements through the range of subscripts
    lrange k start end from 0 or - 1
    8.4llen: length of statistical set
    llen k
    8.5 lpop: delete an element on the left
    lpop k
    8.6 rpop: delete an element on the right
    rpop k

  6. Common commands of type set
    9.1 sadd: used to add elements to the set

    9.2 smembers: used to view all members of the set

    9.3 sismember: judge whether this element is included

    9.4 srem: delete one of the elements

    9.5 scar: Statistics set length

    9.6 sunion: take the union of two sets
    sunion k1 k2
    9.7 sinter: take the intersection of two sets
    sinter k1 k2

  7. Common commands of Zset type
    10.1 zadd: add element to zset set
    10.2 zrevrange: reverse order query
    10.3 zrem: remove an element
    10.4 zcard: length of statistical set
    10.5 zscore: get score

node1:6379> zadd zset1 10000 hadoop 2000 hive 5999 spark 3000 flink
(integer) 4
node1:6379> zrange zset1 0 -1
1) "hive"
2) "flink"
3) "spark"
4) "hadoop"
node1:6379> zrange zset1 0 -1 withscores
1) "hive"
2) "2000"
3) "flink"
4) "3000"
5) "spark"
6) "5999"
7) "hadoop"
8) "10000"
node1:6379> zrange zset1 0 2 withscores
1) "hive"
2) "2000"
3) "flink"
4) "3000"
5) "spark"
6) "5999"
node1:6379> zrevrange zset1 0 -1
1) "hadoop"
2) "spark"
3) "flink"
4) "hive"
node1:6379> zrevrange zset1 0 2
1) "hadoop"
2) "spark"
3) "flink"
node1:6379> zrevrange zset1 0 2 withscores
1) "hadoop"
2) "10000"
3) "spark"
4) "5999"
5) "flink"
6) "3000"
node1:6379> zadd zset1 4000.9 oozie
(integer) 1
node1:6379> zrange zset1 0 -1
1) "hive"
2) "flink"
3) "oozie"
4) "spark"
5) "hadoop"
node1:6379> zrange zset1 0 -1 withscores
 1) "hive"
 2) "2000"
 3) "flink"
 4) "3000"
 5) "oozie"
 6) "4000.9000000000001"
 7) "spark"
 8) "5999"
 9) "hadoop"
10) "10000"
node1:6379> zrem zset1 oozie
(integer) 1
node1:6379> zrange zset1 0 -1 withscores
1) "hive"
2) "2000"
3) "flink"
4) "3000"
5) "spark"
6) "5999"
7) "hadoop"
8) "10000"
node1:6379> zcard zset1
(integer) 4
node1:6379> zscore zset1 flink
  1. Common commands of type bitmap
    11.1 construct bitmaps through the storage space of a string object, and each bit is represented by 0 and 1.
    11.2 setbit: modify the value of a bit
    setbit bit1 0 1
    11.3 bitcount: count the number of a
    bitcount bit1 0 10: how many ones are there in 0-80
    11.4 bitop: and or not operation
    bitop and/or/xor/not
  2. hyperloglog: similar to set, it is used with a large amount of data and has a certain error rate
    12.1 pfadd: add element
    12.2 pfcount: number of Statistics

    12.3 pfmerge: Consolidation
  3. Code construction
    13.1 adding dependencies
                    <!--    <verbal>true</verbal>-->

13.2 building connections

package bigdata.redis;

import org.junit.After;
import org.junit.Before;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;

public class JedisClientTest {
    Jedis jedis=null;//Ordinary jedis object
    public void getConnectipn() {
        //Method 1: directly instantiate and specify the server address: machine port
        //jedis=new Jedis("node1",6379);
        //Method 2: build thread pool
        //Configuration object of thread pool
        JedisPoolConfig config=new JedisPoolConfig();
        config.setMaxTotal(10);//Maximum number of connections to the thread pool
        JedisPool jedisPool=new JedisPool(config,"node1",6379);
        //Get connection from thread pool
 //Close connection
    public void  closeconnect(){

13.3 five types of operations are commonly used:
13.3.1 string operation:

  public void StringTest(){
     jedis.set("name", "lihua");
     jedis.set("s2", "10");
     Boolean s2 = jedis.exists("s2");
     jedis.expire("s2", 10);
     while (true){

13.3.2 hash operation:

   public void hashTest(){
     jedis.hset("hash1", "k1", "hash Is the best");
     Map map=new HashMap();
     map.put("name", "lihua");
     map.put("age", "26");
     map.put("sex", "nan");
     System.out.println(jedis.hmget("m11", "age","sex"));
     Map<String, String> hgetAll = jedis.hgetAll("m11");

13.3.3 list type:

 public  void listTest(){
     jedis.lpush("list1", "3","2","1","0");

13.3.4 set type:

  public  void  setTest(){
     jedis.sadd("set1", "1","2","3","1","9","10","5","3");

13.3.5 zset type

 public  void  zsetTest(){
     jedis.zadd("zset11", 0,"hadoop");
     jedis.zadd("zset11", 30,"hive");
     jedis.zadd("zset11", 40,"flash");
     jedis.zadd("zset11", 20,"happy");
     jedis.zadd("zset11", 100,"hp");
     System.out.println(jedis.zrange("zset11",0, -1));//Sort output
     System.out.println(jedis.zrangeWithScores("zset11", 0,-1));
     System.out.println(jedis.zrevrange("zset11", 0, -1));
  1. Data storage design:
    14.1 how does data storage ensure data security?
    Disk storage: large space, relatively high security, but relatively poor read-write performance; Data security: Replica mechanism (hardware replica: RAIDI, software replica: HDFS block)
    Memory storage: data is stored in memory, with relatively high reading and writing, small space and instability; Data security: the operation log of memory, which appends the changes of memory to the file written to disk or stores the data of the whole memory to disk
    14.2 how to ensure the security of HDFS data?
    The linux disk is through the copy mechanism.
    14.3 how to ensure the security of HDFS metadata?
    The metadata is stored in the memory of the namenode or the fsimage file on the disk of the namenode
    Fault restart: record the changes of memory metadata in the edits file
    Every time you start namenode, merge the edits file with the fsimage file to recover the memory metadata
  2. Persistence of Redis
    15.1 each time redis writes to memory, it synchronizes the data to the disk. If it is restarted, it reloads the data from the disk to memory to provide read and write.
    15.2 RDB scheme: the default persistence scheme of redis: within a certain period of time, if the data in redis memory is updated for a certain number of times, take a full snapshot of all data of the whole redis, and store the file on the hard disk. The new snapshot covers the old snapshot. The snapshot is a full snapshot, which is basically consistent with the memory.
    Manual trigger: data / dump. In redis directory rdb
    Command: save (blocking) or bgsave
    Automatic trigger: take a snapshot according to the number of updates within a certain time.
    There are corresponding configurations in the configuration file:
    There are three groups by default. If there is only one group, it will lead to data loss in different scenarios. For different read and write speeds, it meets the data saving strategy in various cases. There is some data loss.

    15.3 AOF design
    Append the operation log of the memory data to a file. When redis fails and restarts, read all the operation logs from the file and recover the data in memory.
    15.3.1 always: every time a new piece of data is added, the synchronous update operation is added to the file
    15.3.2 appendfsync everysec: write the operation of data in memory asynchronously to the file every second. There is a risk of data loss, with a maximum loss of 1s
    15.3.3 appendfsync no: handed over to the operating system and not controlled by redis

Keywords: Database Redis Cache

Added by Hangston on Mon, 14 Feb 2022 11:56:27 +0200