Elastricearch's fragmentation principle of the second bullet

Recently, a lot of small partners want to see the backstage message ES Later on , Today it comes , I don't say much nonsense , Direct delivery of dry goods .

First let's get to know  ES  On the concept of fragmentation in the process of construction ：ES  Support  PB  Level full text search , When the amount of data on the index is too large ,ES  Split the data on an index into different data blocks through horizontal splitting , The storage is distributed across multiple servers , The split database block is called a partition . Sharding is a fully functional search engine , It has the ability to use all the resources on a node . This is similar to  MySQL  Library sub-table , When writing data in a multi segmented index , Determine which partition to write specifically by path origin , So when creating an index, you need to specify the number of primary partitions , And once the number of main partitions is determined, it cannot be modified （ The reasons are described below ）, Copy fragmentation can be dynamically modified .

So what is the main piece ？ What is replica fragmentation ？

• shard( The primary shard )： What we mentioned above is actually the main partition , The main partition is the container of data , The document is saved in the main partition , The main partition is allocated to each node in the cluster . Every shard It's all one lucene index.

• replica（ A copy of the shard ）： A copy is a piece of  Copy , Synchronously store the data content of the main partition . In order to achieve high availability ,Master  Nodes avoid placing primary and replica shards on the same node , So the maximum number of slices of the copy is  N-1（ among  N  For the node number ）.

The number of partitions and the number of copies can be determined by creating an index  Settings  To configure the ,ES  The default is to create an index  5  Main segments ,  And create a replica tile for each tile .

PUT /myIndex
{
 "settings" : {
 "number_of_shards" : 5,
 "number_of_replicas" : 1
 }
}


Why do you want to slice ？ Why should we distinguish between the main and the auxiliary parts ？

ES  The index is improved in scale and performance through fragmentation , With  shard  You can scale horizontally , Storing more data , Spread search and analysis across multiple servers , Improve throughput and performance .

Read operations ： The search and return data can be processed by the master partition or replica partition at the same time , So the more copies you have, the more pieces you have , The higher the throughput will be ;

The reason for distinguishing between the main and auxiliary parts ： Any server can fail or go down at any time , here shard It might be lost , If for each  shard  establish  replica  copy , When  shard  When the server is not available ,replica  Backup services can be provided , That's the guarantee  ES  In the case of losing a node, no data is lost .

Illustrate with examples

Let's say the node  Node1  Service down or network unavailable , Then the primary partition on the primary node  S0  It's not available . Fortunately, there are two other nodes that work , At this time  ES  A new primary node will be re elected , And there is what we need on these two nodes  S0  All data for . We will  S0  Copy sharding is promoted to primary sharding , This process of ascending the master shard occurs instantaneously . The state of the cluster will be zero  Yellow. Why is our cluster state  Yellow  instead of  Green  Well ？ Although we have all  2  Main segments , But at the same time, each primary partition needs to be divided into two copies , There is only one copy of the shard . So the cluster cannot be  Green  The state of . If we also close  Node2 , Our program can still run without losing any data , because  Node3  A copy is kept for each shard . If we reboot  Node1 , Clusters can shard missing copies for distribution again , The state of the cluster will return to its normal state . If  Node1  Still have before shard , It will try to reuse them , Just then  Node1  The shards on nodes are no longer primary shards but replica Shards , If there is changed data during the period, you only need to copy the modified data file from the main partition .

Knowledge points supplement ：

The cluster state passes   green , yellow , red   To mark ：

• green ： The cluster is healthy and in good condition , All functions are complete and normal , All shards and copies work .

• yellow ： Alert status , All main segments function normally , But at least one copy doesn't work . At this time, the cluster can work normally , But high availability can be affected to some extent .

• Red ： The cluster is not working properly . One or some tiles and their copies are not available , At this time, the query operation of the cluster can also be executed , But the result will be inaccurate . Error will be reported for write requests allocated to this partition , It will eventually lead to the loss of data .

How to write index ： The synchronization principle of primary partition and replica partition

Index can only be written on the main partition , Then synchronize to the replica sharding . Write operations ： New to the document 、 Index and delete requests , It must be completed above the main shard before it can be copied to the relevant replica shard .

As shown in the figure , The picture above shows 3 Nodes ,4 Main segments ,8 Two copies shard , A piece of data  ES  According to what rules is it written on a specific segment ？ First of all, it's not going to be random , Otherwise we won't know where to look when we get the document in the future . actually , This process is determined by the following formula ：

shard = hash(routing) % number_of_primary_shards

Routing  It's a variable value , The default is documentation

_id

, It can also be set to a custom value .Routing  adopt  Hash  The function generates a number , And then this number over here  number_of_primary_shards （ Number of main slices ） And then we get the remainder . This is in  0  To  number_of_primary_shards-1  The remainder between , That's where we're looking for the shard of the document . This explains why we need to determine the number of primary partitions when we create an index and never change it . Because if the quantity changes , Then all previous routing values will be invalid , The document was never found . Because in  ES  Each node in the cluster knows the storage location of documents in the cluster through the above calculation formula , So each node has the ability to process read and write requests . After a write request is sent to a node , This node is the coordination node mentioned earlier , The coordination node will calculate which partition to write to according to the routing formula , Then forward the request to the main shard node of the shard .

ES  In order to improve the ability of writing, this process is written concurrently , At the same time, in order to solve the problem of data conflict in the process of concurrent writing ,ES  Control by optimistic lock , There is one for each document  

_version

 （ edition ） Number , The version number is incremented when the document is modified . Once all the replica shards are successfully written, they will report success to the coordination node , The coordination node reports success to the client .

Illustrate with examples

If the data is redundant through the routing calculation formula, the value is  shard=hash(routing)%4=0.

The specific process is as follows ：

• Client to  ES1  node （ Coordinate nodes ） Send write request , Through the route calculation formula, the value is  0, Then the current data should be written to the main partition  S0  On .

• ES1  The node forwards the request to  S0  The node where the main partition is located  ES3,ES3  Accept request and write to disk .

• Copy the data to two copies at the same time  R0  On , Among them, optimistic concurrency is used to control data conflicts . Once all copy shards are reported successful , The node  ES3  Success will be reported to the coordination node , Coordinate nodes （ES1） Report success to client .

The more copies are divided, the better ？

The answer, of course, is  no , There are two reasons ：

• （1） Multiple  replica  It can improve the throughput and performance of search operations , However, if you only add more replica fragmentation on the same number of nodes, it will not improve the performance , Because each shard gets fewer resources from the node , At this time, you need to add more hardware resources to improve throughput .

• （2） More replica fragmentation increases data redundancy , Data integrity guaranteed , However, according to the principle of interaction between the upper main and sub segments , Data synchronization between partitions will occupy a certain amount of network bandwidth , Affect efficiency , So the number of slices and copies of index is not the more the better .

summary ：

• The purpose of data fragmentation is to improve the capacity of data that can be processed and to expand it horizontally , Making replicas for sharding is to improve the stability of the cluster and the amount of concurrency .

• Copy is multiplication , The more consumption, the greater , But the more secure . Fragmentation is division , More segments , The less and more fragmented the single slice data is .

• More copies , The higher the availability of the cluster , But because each slice is equivalent to a  Lucene  The index file of , It will occupy a certain file handle 、 Memory and  CPU.

o  Q  Will continue to update  java  Practical articles , If you have a different opinion or a better one  idea, Welcome to contact ah  Q.

【 o  Q  Say code 】, Notable official account

The style of the article is changeable , The pictures are easy to understand , The story is vivid and interesting , Let's talk about technology ！