Introduction to yottastore, a new generation of storage engine for VPC TCE cos

Object storage （Cloud Object Storage,COS） It is a distributed storage service for storing massive files provided by Tencent's proprietary cloud , Users can store and view data at any time through the network . All users pass COS Can be used with high scalability 、 Low cost 、 Highly reliable and secure data storage services .

The Internet has entered the era of data explosion ZettaByte Time , The most basic service of cloud computing —— Storage , Facing great challenges . With the explosive growth of user data , Business needs are getting higher and higher , The traditional distributed storage system has gradually failed .YottaStore Should be “ cloud ” born ,YottaStore It is Tencent's proprietary cloud object storage （COS） A new generation of ultra large scale cloud storage engine .

YottaStore Support Any number of copies and erasure codes coexist in redundancy mode . Greatly improve system availability 、 Reliability and performance , Dramatically lower storage costs . A single cluster with millions of nodes The system is highly autonomous , Fully automated Data patrol for 、 Repair 、 equilibrium 、 Recycling 、 Expansion and contraction capacity 、 Request traffic scheduling and fault aware processing , Maximize the service quality of the system , Greatly reduce the operation and maintenance costs .YottaStore At present, it has been launched in several clusters of Tencent public cloud , The operation quality is far beyond SLA.

The overall architecture

YottaStore The system adopts layered design in the main business , Provides better scalability 、 Maintainability and iterativeness , It is mainly divided into the following layers ：

besides ,YottaStore The system also designs several subsystems to complete other functions , Comprehensively improve system availability 、 Reliability and maintainability ： Space allocation subsystem 、 Check and repair subsystem 、 Data equalization subsystem 、 Health management subsystem 、 Cluster management subsystem .

Data redundancy storage scheme

Coding redundant storage mode

Erasure code It's a coding technique , It can be used K Raw data , Code generation N Coded data , And can pass N Any of the shares K Copy of the data , Restore raw data .EC Technology is applied to distributed storage system , Under the premise of ensuring the same data reliability , Significantly reduce data redundancy , Reduce storage costs .

Assume that the encoding method is K-N, Replica redundancy is N/K, The original data is encoded to form the same size N Copy of the data ,N No more than... Pieces of data are lost N-K Copy of the data , The original data can be recovered . give an example 11-21 Coding method of , Assume that the original data size is 11MB, After coding, it is formed 21 The share size is 1MB The data of , this 21 Any of the data is lost 10 Copy of the data , Through other 11 Copies of data to recover .

Streaming online EC

The traditional use of EC Redundant storage system , Data is often stored in the form of multiple copies , After accumulating a certain amount of data , Then the offline code is EC Redundant storage . and YottaStore The system has realized online EC, User data is directly processed in the access layer EC Encode and write directly to the storage node .

Compared to offline EC, On-line EC The advantages are obvious ：

1. nothing buffer Layer storage , No secondary index ： Simplify system design , Improve data reliability , Reduce storage costs .
2. Delete space real-time recycling ： Traditional offline EC The design of the , After the user deletes the data , It will leave holes in the data encoding block , Reclaiming this void space requires rewriting the data after the void rate reaches a certain level , Space recycling is not timely , The recycling process also wastes a lot IO.YottaStore Use streaming online EC Design , Cooperate with self-developed 、 Stand alone storage engine that directly operates block devices , It realizes the near real-time recycling of deleted space . This significantly reduces storage costs , It also avoids the need for rewriting operations IO, Affect online throughput capacity and service stability .
3. Write performance improvement ： Significant savings in write operations IO And network traffic amplification , Increase write throughput .
4. Read performance improvement ： Avoid offline EC Secondary index read and reconstruction read amplification .

Flexible replica redundancy scheme

YottaStore Multiple storage modes coexist in the same cluster , Can support any at the same time EC Encoding and multi copy redundancy mode . It avoids the waste of resources caused by setting up multiple clusters for different redundancy , And the redundancy can be flexibly selected according to the actual scene , Both availability 、 Reliability and cost .

Technical highlights

Very large scale clusters （ Single cluster millions of nodes ）

YottaStore Through the integration of the traditional distributed storage system Master Carried out all-round 、 Complete disassembly , Breaking through a single point Master Bottleneck , Metadata hierarchical autonomy can effectively reduce the amount of metadata , Internal logic splits subsystems by function , A single cluster can reach millions of nodes .

Top N Return & Backup Request Mechanism

YottaStore use Top N Return To handle write requests , It can effectively avoid the request failure caused by the failed node , Improve usability . meanwhile ,Top N Return It can also effectively discard long tail requests , Improve service quality .

YottaStore use Backup Request + Top N Return To reduce the read delay . In theory, you just need to read K One copy can decode the original data , To improve read availability , Reduce read latency , May be increased appropriately 1-2 A parallel Backup Request, Any of them K The data decoding can be completed when the requests are returned successfully , Back to the user .

Adaptive flow control

YottaStore A routing policy module is built in the access module of , Refine each request to the health status of the storage node 、 Operating status and IOPS Real time evaluation of times , So as to achieve refinement 、 Autonomous territory 、 Adaptively control flow in and out , Ensure that the service is robust and stable .

YottaStore The storage node of adopts hierarchical token bucket design , The traffic with higher priority has been reached , bandwidth / The more storage resources are prioritized, the better .

Global load balancing

According to the physical resource usage of the system , Adopt appropriate strategies , adjustment Vlet The distribution of , Excellent global data balancing strategy , This makes the average disk utilization up to 95% above .

Multi level patrol repair mechanism

YottaStore The system introduces multi-level patrol inspection （ Stand alone storage engine patrol 、 Data consistency inspection 、 Data integrity inspection ） Mechanism , Ensure that data exceptions are sensed and fixed at the first time , Greatly improves data reliability .YottaStore The disk level automatic operation and maintenance and data repair linkage mechanism is realized on the data storage node .

Fast hot upgrade without user awareness

YottaStore Innovatively introduced the concepts of management area and isolation level , On the basis of management area and isolation level ,YottaStore Realize the rapid hot upgrade without user perception .

Industrial consistency protocol library SCAL

SCAL Is an industrial level implementation of distributed consistency protocol library . The whole library is implemented in a pure asynchronous way , And made a series of innovative designs leading the industry and optimized implementations highly consistent with the business .

Conclusion

All rivers run into sea , If there is room, there is room . Object storage （Cloud Object Storage,COS） A new generation of storage engines YottaStore, In addition, it means that the storage scale can reach “ infinity ” The expectations of the , We also hope that our users can store them “Yotta”, Can feel at ease 、 don 't worry . Tencent's proprietary cloud storage team will continue to work hard , With the greatest enthusiasm , The most solid technical foundation , To escort the vast number of users , Conquer the vast star sea of scientific and technological innovation ！

-END-