1. Advantages of reducing redis memory usage
1. Help reduce the time required to create and load snapshots
2. Improving the efficiency of loading and rewriting AOF files
3. Shorten the time required to synchronize from the server
4. Allowing redis to store more data without adding additional hardware
2. Short structure
Redis provides a set of configuration options for lists, collections, hashes, and ordered collections that allow redis to store shorter structures in a more economical way.
2.1, ziplist compressed list (list, hash, sequel and)
Normally used storage
When lists, hashes, ordered sets are short in length or small in size, redis will use a compact storage method called ziplist to store these structures.
ziplist is an unstructured representation of three different types of objects: list, hash and ordered set. It stores data in a serialized way. The serialized data need to be decoded every time it is read and coded every time it is written.
Differences between bi-directional lists and compressed lists:
In order to understand that compressed lists save more memory than other data structures, we take the list structure as an example for in-depth study.
Typical bi-directional lists
In a typical two-way list, each value has a node representation. Each node will have a pointer to the previous node and the latter node of the list, as well as a pointer to the string value contained in the node.
Each node contains string values that are stored in three parts. Includes the length of the string, the number of bytes remaining in the string value, and the string itself ending with empty characters.
Examples:
If a node stores an'abc'string, conservatively estimates the additional overhead of 21 bytes on a 32-bit platform (three pointers + two int + null characters: 3*4 + 2*4 + 1 = 21)
An example shows that storing a 3-byte string requires at least 21 bytes of additional overhead.
ziplist
Compressed lists are sequences of nodes, each containing two lengths and a string. The first length records the length of the previous node (used to traverse the compressed list backwards and forwards); the second is the length of the current point of the record book; and the stored string.
Examples:
The storage string'abc', both lengths can be stored in 1 byte, resulting in an additional overhead of 2 bytes (two lengths, i.e. 1 + 1 = 2)
Conclusion:
Compressed lists reduce additional overhead by avoiding storing additional pointers and metadata.
Configuration:
1 #list 2 list-max-ziplist-entries 512 #Represents the maximum number of elements allowed to be included 3 list-max-ziplist-value 64 #Represents the maximum allowable storage volume for compressed nodes 4 #hash #When any limit is exceeded, ziplist storage will not be used 5 hash-max-ziplist-entries 512 6 hash-max-ziplist-value 64 7 #zset 8 zset-max-ziplist-entries 128 9 zset-max-ziplist-value 64
Test list:
1. Establishing test.php file
1 #test.php 2 <?php 3 $redis=new Redis(); 4 $redis->connect('192.168.95.11','6379'); 5 for ($i=0; $i<512 ; $i++) 6 { 7 $redis->lpush('test-list',$i.'-test-list'); #to test-list Push in 512 data 8 } 9 ?>
There are 512 pieces of data in the test-list at this time, which does not exceed the limit in the configuration file.
2. Push another data into the test-list
At this time, test-list contains 513 pieces of data, which is larger than 512 pieces Limited in configuration file. The index will abandon ziplist storage mode and adopt its original linkedlist storage mode.
Hashing is the same as ordered sets.
2.2, intset integer set (set)
The precondition is that all member s contained in a set can be resolved into decimal integers.
Storing collections in an ordered array can not only reduce memory consumption, but also improve the execution speed of collections operations.
Configuration:
1 set-max-intset-entries 512 #Limited set member Number, no more than intset storage
Testing:
Establish test.php file
1 #test.php 2 <?php 3 $redis=new Redis(); 4 $redis->connect('192.168.95.11','6379'); 5 for ($i=0; $i<512 ; $i++) 6 { 7 $redis->sadd('test-set',$i); #Give Set test-set Insert 512 member 8 } 9 ?>
2.3. Performance issues
Despite lists, hashes, ordered sets, and collections, when the restrictions are exceeded, they will be converted to more typical underlying structure types. Because as the size of compact structures increases, the speed of operating these structures will become slower and slower.
Testing:
# list will be used for representative testing
Test ideas:
1. Push 50,000 pieces of data into the test-list under the default configuration to see the time needed; then use rpoplpush to push all the test-list data into the new list-new to see the time needed
2. Modify the configuration, list-max-ziplist-entries 100000, and then perform the same operation as above.
3. Contrast the time and draw a conclusion
Testing under default configuration:
1. Insert data and view time
1 #test1.php 2 <?php 3 header("content-type: text/html;charset=utf8;"); 4 $redis=new Redis(); 5 $redis->connect('192.168.95.11','6379'); 6 $start=time(); 7 for ($i=0; $i<50000 ; $i++) 8 { 9 $redis->lpush('test-list',$i.'-aaaassssssddddddkkk'); 10 } 11 $end=time(); 12 echo "The insertion time is:".($end-$start).'s'; 13 ?>
The result took four seconds
2. Executing the corresponding commands and viewing time-consuming
1 #test2.php 2 <?php 3 header("content-type: text/html;charset=utf8;"); 4 $redis=new Redis(); 5 $redis->connect('192.168.95.11','6379'); 6 $start=time(); 7 $num=0; 8 while($redis->rpoplpush('test-list','test-new')) 9 { 10 $num+=1; 11 } 12 echo 'The number of executions is:'.$num."<br/>"; 13 $end=time(); 14 echo "Time-consuming:".($end-$start).'s'; 15 ?>
Change the test under the configuration file
1. Modify the configuration file first
list-max-ziplist-entries 100000 # Modify this value a little larger to better highlight the impact on Performance
list-max-ziplist-value 64 # This value is unchanged
2. Insert data
Execute test1.php
The result is that it takes 12 seconds.
3. Executing the corresponding commands and viewing time-consuming
Execute test2.php
The results were as follows: execution times: 50,000, time-consuming 12 seconds.
Conclusion:
In this machine, there is a difference of 8 seconds when 50,000 data are tested. In the case of high concurrency, long compressed lists and large integer sets will not have any optimization, but will reduce performance.
3. Sheet structure
The essence of fragmentation is to divide the data into smaller parts based on simple rules, and then decide where to send the data according to the part to which the data belongs. Many databases use this technology to expand storage space and increase the amount of load they can handle.
Considering the previous discussion, it is not difficult to find the importance of fragmentation structure for redis. So we need to make appropriate adjustments in the configuration file about ziplist and intset.
3.1. Piecewise hashing
#ShardHash.class.php
1 <?php 2 class ShardHash 3 { 4 private $redis=''; #storage redis object 5 /** 6 * @desc Constructor 7 * 8 * @param $host string | redis Host 9 * @param $port int | port 10 */ 11 public function __construct($host,$port=6379) 12 { 13 $this->redis=new Redis(); 14 $this->redis->connect($host,$port); 15 } 16 17 /** 18 * @desc Calculate the slice ID of a key 19 * 20 * @param $base string | Base hash 21 * @param $key string | Key names to be stored in a fragmented hash 22 * @param $total int | Estimated total number of non-digital fragments 23 * 24 * @return string | Return the splitting key 25 */ 26 public function shardKey ($base,$key,$total) 27 { 28 if(is_numeric($key)) 29 { 30 $shard_id=decbin(substr(bindec($key),0,5)); #take $key Binary high five decimal values 31 } 32 else 33 { 34 $shard_id=crc32($key)%$shards; #Finding Remainder Module 35 } 36 return $base.'_'.$shard_id; 37 } 38 39 /** 40 * @desc Fragmented hash hset operation 41 * 42 * @param $base string | Base hash 43 * @param $key string | Key names to be stored in a fragmented hash 44 * @param $total int | Total number of predicted elements 45 * @param $value string/int | value 46 * 47 * @return bool | Is hset successful 48 */ 49 public function shardHset($base,$key,$total,$value) 50 { 51 $shardKey=$this->shardKey($base,$key,$total); 52 return $this->redis->hset($shardKey,$key,$value); 53 } 54 55 /** 56 * @desc Fragmented hash hget operation 57 * 58 * @param $base string | Base hash 59 * @param $key string | Key names to be stored in a fragmented hash 60 * @param $total int | Total number of predicted elements 61 * 62 * @return string/false | Successful return of value 63 */ 64 public function shardHget($base,$key,$total) 65 { 66 $shardKey=$this->shardKey($base,$key,$total); 67 return $this->redis->hget($shardKey,$key); 68 } 69 70 } 71 72 $obj=new ShardHash('192.168.95.11'); 73 echo $obj->shardHget('hash-','key',500); 74 ?>
Hash splitting calculates the splitting key ID based on the base key and the key contained in the hash, and then splices it with the base key to form a complete splitting key. In the execution of hset and hget and most hash commands, key (field) needs to be processed by shardKey method before the next operation can be carried out.
3.2. Piecewise Sets
How to construct a fragmented collection to save more memory and make it more powerful? The main idea is to convert the data stored in the collection into decimal data as far as possible without changing its original function. As mentioned earlier, when all members of a collection can be parsed into decimal data, intset storage will be used, which not only saves memory, but also improves response performance.
Example:
If you want a large website, you need to store the unique user visits every day. Then the unique identifier of the user can be converted into decimal digits and stored in a fragmented set.
#ShardSet.class.php
1 <?php 2 class ShardSet 3 { 4 private $redis=''; #storage redis object 5 /** 6 * @desc Constructor 7 * 8 * @param $host string | redis Host 9 * @param $port int | port 10 */ 11 public function __construct($host,$port=6379) 12 { 13 $this->redis=new Redis(); 14 $this->redis->connect($host,$port); 15 } 16 17 /** 18 * @desc Calculate the splitting keys based on the base keys and hashed included keys 19 * 20 * @param $base string | Base hash 21 * @param $key string | Key names to be stored in a fragmented hash 22 * @param $total int | Estimated total number of fragments 23 * 24 * @return string | Return the splitting key 25 */ 26 public function shardKey ($base,$member,$total=512) 27 { 28 $shard_id=crc32($member)%$shards; #Finding Remainder Module 29 return $base.'_'.$shard_id; 30 } 31 32 /** 33 * @desc Calculating the Daily Access of Unique Users 34 * 35 * @param $member int | User Unique Identifier 36 * 37 * @return string | ok Represents count plus 1 false to indicate that the user has accessed no more than 1 today 38 */ 39 public function count($member) 40 { 41 $shardKey=$this->shardKey('count',$member,$total=10); #$totla Make it smaller for testing 42 $exists=$this->redis->sismember($shardKey,$member); 43 if(!$exists) #judge member Have you visited today? 44 { 45 $this->redis->sadd($shardKey,$member); 46 $this->redis->incr('count'); 47 $ttl1=$this->redis->ttl('count'); 48 if($ttl1===-1) 49 $this->redis->expireat('count',strtotime(date('Y-m-d 23:59:59'))); #Set expiration time 50 $ttl2=$this->redis->ttl($shardKey); 51 if($ttl2===-1) 52 { 53 $this->redis->expireat("$shardKey",strtotime(date('Y-m-d 23:59:59'))); #Set expiration time 54 #echo $shardKey; #Test usage 55 } 56 #echo $shardKey; #Test usage 57 return 'ok'; 58 } 59 return 'false'; 60 } 61 62 63 } 64 65 66 67 $str=substr(md5(uniqid()), 0, 8); #Remove the top eight 68 #take $str Unique identifier for customers 69 $str=hexdec($str); #Converting hexadecimal to decimal 70 $obj=new ShardSet('192.168.95.11'); 71 $obj->count($str); 72 73 ?>
4. Packing information into storage bytes
Combining with the previous fragmentation technology, string fragmentation structure is adopted to store information for a large number of continuous ID users.
A fixed-length string is used to allocate n bytes for each ID to store the corresponding information.
Next, we will use the examples of storage user countries and provinces to illustrate:
If a user needs to store the information of China and Guangdong Province and use utf8 character set, it will consume at least 5*3=15 bytes. If the number of users of the website is large, it will take up a lot of resources. Next, we adopt the method that each user only needs two bytes to complete the storage of information.
Specific ideas and steps:
1. First of all, we set up corresponding information forms for the information of the country and the provinces in each country.
2. The establishment of the Information Form also means that each country and province has its corresponding index number.
3. Looking at this, you should all think of it. That is to say, two indexes are used as information stored by users. However, it should be noted that we still need to deal with these two indexes accordingly.
4. Treat index as ASCII code and convert it to corresponding characters specified by ASCII (0-255)
5. Find out the storage location of users by using the fragmentation technology mentioned above and the fixed-length fragmentation string structure (a string in redis can not exceed 512M)
6. Implementing the write-in and take-out of information (getrange, setrange)
Implementation code:
#PackBytes.class.php
1 <?php 2 #Packed storage bytes 3 #Storing User Country and Provincial Information 4 class PackBytes 5 { 6 private $redis=''; #storage redis object 7 /** 8 * @desc Constructor 9 * 10 * @param $host string | redis Host 11 * @param $port int | port 12 */ 13 public function __construct($host,$port=6379) 14 { 15 $this->redis=new Redis(); 16 $this->redis->connect($host,$port); 17 } 18 19 /** 20 * @desc Processing and caching national provincial data 21 * @param $countries string | Type I data, country string 22 * @param $provinces Two-dimensional array | Category II data, provincial arrays 23 * @param $cache 1/0 | Whether to use caching, default 0 does not use 24 * 25 * @return array | Return total data 26 */ 27 public function dealData($countries,$provinces,$cache=0) 28 { 29 if($cache) 30 { 31 $result=$this->redis->get('cache_data'); 32 if($result) 33 return unserialize($result); 34 } 35 $arr=explode(' ',$countries); 36 $areaArr[]=$arr; 37 $areaArr[]=$provinces; 38 $cache_data=serialize($areaArr); 39 $this->redis->set('cache_data',$cache_data); 40 return $areaArr; 41 } 42 43 /** 44 * @desc Converting Specific Information into Coded Information by Table Index 45 * 46 * @param $countries,$provinces，$cache| Reference to dealData method 47 * @param $country string | Specific information - country 48 * @param $province string | Specific Information - Provinces 49 * 50 * @return string | Returns the encoding information of the transformation 51 */ 52 public function getCode($countries,$provinces,$country,$province,$cache=0) 53 { 54 $dataArr=$this->dealData($countries,$provinces,$cache=0); 55 56 $result=array_search($country, $dataArr[0]); #Find if the array contains data1 57 if($result===false) #Judgment of existence 58 return chr(0).chr(0); #Returns the initial value if nonexistent 59 $code=chr($result); 60 $result=array_search($province, $dataArr[1][$country]); #Find if the array contains data2 61 if($result===false) 62 return $code.chr(0); 63 return $code.chr($result); #Return correspondence ASCII(0~255)Characters specified 64 } 65 66 /** 67 * @desc Calculating the Relevant Location Information of the Coded Data Stored by the User 68 * 69 * @param $userID int | User ID 70 * 71 * @return array | Returns an array containing the fragmented ID when the data is stored, and the storage location (offset) that belongs to the user. 72 */ 73 public function savePosition($userID) 74 { 75 $shardSize=pow(2, 3); #Size of each fragment 76 $position=$userID*2; #user Ranking 77 $arr['shardID']=floor($position/$shardSize); #Fragmentation ID 78 $arr['offset']=$position%$shardSize; #Offset 79 return $arr; 80 } 81 82 /** 83 * @desc | Integration method to store coded information in the corresponding position of string in redis 84 * 85 * @param $userID int | User ID 86 * @param $countries string | Type I data, country string 87 * @param $provinces Two-dimensional array | Category II data, provincial arrays 88 * @param $country string | Specific information - country 89 * @param $province string | Specific Information - Provinces 90 * @param $cache 1/0 | Whether to use caching, default 0 does not use 91 * 92 * @return Successful return to write location/failed false 93 */ 94 public function saveCode($userID,$countries,$provinces,$country,$province,$cache=0) 95 { 96 $code=$this->getCode($countries,$provinces,$country,$province,$cache=0); 97 $arr=$this->savePosition($userID); #Storage of relevant location information 98 return $this->redis->setrange('save_code_'.$arr['shardID'],$arr['offset'],$code); 99 } 100 101 /** 102 * @desc Access to user-specific country and province information 103 * 104 * @param $userID int | User ID 105 * 106 * @return array | Returns an array of country and province information 107 */ 108 public function getMessage($userID) 109 { 110 $position=$this->savePosition($userID); 111 $code=$this->redis->getrange('save_code_'.$position['shardID'],$position['offset'],$position['offset']+1); 112 $arr=str_split($code); 113 $areaArr=$this->dealData('', '',$cache=1); #Using cached data 114 $message['country']=$areaArr[0][ord($arr[0])]; 115 $message['province']=$areaArr[1][$message['country']][ord($arr[1])]; 116 return $message; 117 } 118 119 } 120 121 header("content-type: text/html;charset=utf8;"); 122 $countries="No China, Japan, Vietnam, Korea, Russia, Pakistan, USA"; 123 $provinces=array( 124 'nothing'=>array('nothing'), 125 'China'=>array('nothing','Guangdong','Hunan','Hubei','Guangxi','Yunnan','Hunan','Hebei'), 126 'Japan'=>array('nothing','Tortoise Sunzi District','Wangba District','Ghost District of Japan','Guizi District','Radish head area'), 127 ); 128 $obj=new PackBytes('192.168.95.11'); 129 /* 130 #Data processing and caching in redis 131 $b=$obj->dealData($countries,$provinces); 132 echo "<pre>"; 133 print_r($b); 134 echo "</pre>";die; 135 */ 136 /* 137 #Storage of user's national and provincial information 138 $country='China'; 139 $province='Guangdong'; 140 $result=$obj->saveCode(0,$countries,$provinces,$country,$province); 141 echo "<pre>"; 142 print_r($result); 143 echo "</pre>"; 144 */ 145 /* 146 #Extract user country provincial information 147 $a=$obj->getMessage(15); 148 echo "<pre>"; 149 print_r($a); 150 echo "</pre>";die; 151 */ 152 153 ?>
Test:
1. The information processed by dealData is information table table.
2,saveCode()
| userID | Country | Province |
| 0 | China | Guangdong |
| 13 | Japan | Tortoise Sunzi District |
| 15 | Japan | Wangba District |
,
3,getMessage()
Reference Books:
Redis Actual Warfare by Josiah Carlson
Translated by Huang Jianhong
(The above are some of my own opinions. If there are any shortcomings or mistakes, please point them out.)
Authors: A leaf follows the wind
Disclaimer: This blog article is original and only represents the views or conclusions I summarized at a certain time in my work and study. When reprinting, please give a link to the original text in a clear place on the article page.