Redis pipeline technology / partition

Redis Pipeline technology

Redis Is a client based - Server models and requests / In response to the protocol TCP service . This means that the request will usually follow the following steps ： The client sends a query request to the server , And listen for socket returns , Usually in blocking mode , Wait for the server to respond . The server processes the command and returns the result to the client .Redis Pipeline technology allows the client to continue sending requests to the server when the server does not respond , Read the responses of all servers for the last time .

example

To see redis The Conduit , Just start redis Instance and enter the following command ：

$(echo -en "PING\r\n SET runoobkey redis\r\nGET runoobkey\r\nINCR visitor\r\nINCR visitor\r\nINCR visitor\r\n"; sleep 10) | nc localhost 6379

+PONG
+OK
redis
:1
:2
:3


In the example above , We use ping Command check redis Is the service available . Then we will runoobkey Is set to redis, And then we get runoobkey Value , And automatically triple the number of visitors . In the returned results , We can see that these commands are submitted to redis service , The most significant advantage of the response pipeline technology of reading all servers for the last time is to improve redis Service performance . Some test data

require 'rubygems' 
require 'redis'
def bench(descr) 
start = Time.now 
yield 
puts "#{descr} #{Time.now-start} seconds" 
end
def without_pipelining 
r = Redis.new 
10000.times { 
 r.ping 
} 
end
def with_pipelining 
r = Redis.new 
r.pipelined { 
 10000.times { 
 r.ping 
 } 
} 
end
bench("without pipelining") { 
 without_pipelining 
} 
bench("with pipelining") { 
 with_pipelining 
}


In the next test , We will use redis Of Ruby Client to support the characteristics of Pipeline Technology , And test the speed-up effect of assembly line technology . In LAN Mac OS X The data of the above simple script executed on the system shows , After opening the pipeline , Round trip latency has improved significantly .

Redis

Partitioning is to divide data into multiple redis The process of example , Therefore, each instance only saves a subset of the key . The advantage of partitioning is achieved by using the total memory of multiple computers , We can build a larger database . Through multiple cores and computers , We can expand our computing power ; Through multiple computers and network adapters , We can expand the network bandwidth . Insufficient partitions redis Some functions of the do not perform well in partitioning ： Typically, multiple key operations are not supported . for example , When two sets map to different redis When an instance , You cannot perform an intersection operation on these two sets . Cannot use multiple keys Redis Business . When using partitions , Data processing is more complex . for example , You need to deal with multiple rdb/aof file , And back up persistent files from multiple instances and hosts . Adding or removing capacity is also complex . majority redis The cluster supports transparent data balancing for adding and deleting nodes at run time , But other systems （ Such as client partition and proxy ） This feature is not supported . However , A technique called pre hardening is helpful .

Partition type

Redis There are two types of divisions . Suppose there are four redis example R0、R1、R2、R3 And multiple keys represent the user , for example user:1 and user:2. For a given key , There are many different ways to choose which instance the key is stored in . let me put it another way , There are different systems that map keys to redis service .

Range partitioning

The simplest way to partition is by range , Mapping objects in a specific range to specific redis example . for example ,ID by 0 To 10000 The user of will be saved to the instance R0,ID by 10001 To 20000 Users of will be saved to R1, And so on . This method is feasible , It can be used in practice . The disadvantage is that there is a mapping table from interval range to instance . This table needs to be managed . meanwhile , It also requires mapping tables for various objects , It's not usually redis Good way .

Hash partition

Another partitioning method is hash partitioning . This applies to any key , And it doesn't need to be an object name ： This form is as simple as the following description ： Use the hash function to convert the key to a number , for example , Use CRC32 hash function . Key in foobar On the implementation CRC32（foobar） Will output a similar 93024922 The integer of . Take the modulus of this integer and convert it to 0 To 3 Number between , Then this integer can be mapped to four redis One example .93024922%4=2, This means that the key foobar It should be kept in R2 In the example . notes ： Modular operation is the remainder of division , In many programming languages % Operator implementation .