当前位置:网站首页>Introduction to yottastore, a new generation of storage engine for VPC TCE cos
Introduction to yottastore, a new generation of storage engine for VPC TCE cos
2022-06-24 17:37:00 【Tencent proprietary cloud】
Object storage (Cloud Object Storage,COS) It is a distributed storage service for storing massive files provided by Tencent's proprietary cloud , Users can store and view data at any time through the network . All users pass COS Can be used with high scalability 、 Low cost 、 Highly reliable and secure data storage services .
The Internet has entered the era of data explosion ZettaByte Time , The most basic service of cloud computing —— Storage , Facing great challenges . With the explosive growth of user data , Business needs are getting higher and higher , The traditional distributed storage system has gradually failed .YottaStore Should be “ cloud ” born ,YottaStore It is Tencent's proprietary cloud object storage (COS) A new generation of ultra large scale cloud storage engine .
YottaStore Support Any number of copies and erasure codes coexist in redundancy mode . Greatly improve system availability 、 Reliability and performance , Dramatically lower storage costs . A single cluster with millions of nodes The system is highly autonomous , Fully automated Data patrol for 、 Repair 、 equilibrium 、 Recycling 、 Expansion and contraction capacity 、 Request traffic scheduling and fault aware processing , Maximize the service quality of the system , Greatly reduce the operation and maintenance costs .YottaStore At present, it has been launched in several clusters of Tencent public cloud , The operation quality is far beyond SLA.
The overall architecture
YottaStore The system adopts layered design in the main business , Provides better scalability 、 Maintainability and iterativeness , It is mainly divided into the following layers :
besides ,YottaStore The system also designs several subsystems to complete other functions , Comprehensively improve system availability 、 Reliability and maintainability : Space allocation subsystem 、 Check and repair subsystem 、 Data equalization subsystem 、 Health management subsystem 、 Cluster management subsystem .
Data redundancy storage scheme
Coding redundant storage mode
Erasure code It's a coding technique , It can be used K Raw data , Code generation N Coded data , And can pass N Any of the shares K Copy of the data , Restore raw data .EC Technology is applied to distributed storage system , Under the premise of ensuring the same data reliability , Significantly reduce data redundancy , Reduce storage costs .
Assume that the encoding method is K-N, Replica redundancy is N/K, The original data is encoded to form the same size N Copy of the data ,N No more than... Pieces of data are lost N-K Copy of the data , The original data can be recovered . give an example 11-21 Coding method of , Assume that the original data size is 11MB, After coding, it is formed 21 The share size is 1MB The data of , this 21 Any of the data is lost 10 Copy of the data , Through other 11 Copies of data to recover .
Streaming online EC
The traditional use of EC Redundant storage system , Data is often stored in the form of multiple copies , After accumulating a certain amount of data , Then the offline code is EC Redundant storage . and YottaStore The system has realized online EC, User data is directly processed in the access layer EC Encode and write directly to the storage node .
Compared to offline EC, On-line EC The advantages are obvious :
- nothing buffer Layer storage , No secondary index : Simplify system design , Improve data reliability , Reduce storage costs .
- Delete space real-time recycling : Traditional offline EC The design of the , After the user deletes the data , It will leave holes in the data encoding block , Reclaiming this void space requires rewriting the data after the void rate reaches a certain level , Space recycling is not timely , The recycling process also wastes a lot IO.YottaStore Use streaming online EC Design , Cooperate with self-developed 、 Stand alone storage engine that directly operates block devices , It realizes the near real-time recycling of deleted space . This significantly reduces storage costs , It also avoids the need for rewriting operations IO, Affect online throughput capacity and service stability .
- Write performance improvement : Significant savings in write operations IO And network traffic amplification , Increase write throughput .
- Read performance improvement : Avoid offline EC Secondary index read and reconstruction read amplification .
Flexible replica redundancy scheme
YottaStore Multiple storage modes coexist in the same cluster , Can support any at the same time EC Encoding and multi copy redundancy mode . It avoids the waste of resources caused by setting up multiple clusters for different redundancy , And the redundancy can be flexibly selected according to the actual scene , Both availability 、 Reliability and cost .
Technical highlights
Very large scale clusters ( Single cluster millions of nodes )
YottaStore Through the integration of the traditional distributed storage system Master Carried out all-round 、 Complete disassembly , Breaking through a single point Master Bottleneck , Metadata hierarchical autonomy can effectively reduce the amount of metadata , Internal logic splits subsystems by function , A single cluster can reach millions of nodes .
Top N Return & Backup Request Mechanism
YottaStore use Top N Return To handle write requests , It can effectively avoid the request failure caused by the failed node , Improve usability . meanwhile ,Top N Return It can also effectively discard long tail requests , Improve service quality .
YottaStore use Backup Request + Top N Return To reduce the read delay . In theory, you just need to read K One copy can decode the original data , To improve read availability , Reduce read latency , May be increased appropriately 1-2 A parallel Backup Request, Any of them K The data decoding can be completed when the requests are returned successfully , Back to the user .
Adaptive flow control
YottaStore A routing policy module is built in the access module of , Refine each request to the health status of the storage node 、 Operating status and IOPS Real time evaluation of times , So as to achieve refinement 、 Autonomous territory 、 Adaptively control flow in and out , Ensure that the service is robust and stable .
YottaStore The storage node of adopts hierarchical token bucket design , The traffic with higher priority has been reached , bandwidth / The more storage resources are prioritized, the better .
Global load balancing
According to the physical resource usage of the system , Adopt appropriate strategies , adjustment Vlet The distribution of , Excellent global data balancing strategy , This makes the average disk utilization up to 95% above .
Multi level patrol repair mechanism
YottaStore The system introduces multi-level patrol inspection ( Stand alone storage engine patrol 、 Data consistency inspection 、 Data integrity inspection ) Mechanism , Ensure that data exceptions are sensed and fixed at the first time , Greatly improves data reliability .YottaStore The disk level automatic operation and maintenance and data repair linkage mechanism is realized on the data storage node .
Fast hot upgrade without user awareness
YottaStore Innovatively introduced the concepts of management area and isolation level , On the basis of management area and isolation level ,YottaStore Realize the rapid hot upgrade without user perception .
Industrial consistency protocol library SCAL
SCAL Is an industrial level implementation of distributed consistency protocol library . The whole library is implemented in a pure asynchronous way , And made a series of innovative designs leading the industry and optimized implementations highly consistent with the business .
Conclusion
All rivers run into sea , If there is room, there is room . Object storage (Cloud Object Storage,COS) A new generation of storage engines YottaStore, In addition, it means that the storage scale can reach “ infinity ” The expectations of the , We also hope that our users can store them “Yotta”, Can feel at ease 、 don 't worry . Tencent's proprietary cloud storage team will continue to work hard , With the greatest enthusiasm , The most solid technical foundation , To escort the vast number of users , Conquer the vast star sea of scientific and technological innovation !
-END-
边栏推荐
- Error reported after NPM I
- Elastic searchable snapshot function (frozen Tier 3)
- Construction scheme of campus network clock system (standardized examination room)
- Snapshot management for elastic cloud enterprise
- [log service CLS] Tencent cloud game battle engine mgobe accesses CLS
- New MySQL 8.0 feature - enhanced logical backup recovery
- Research on clock synchronization performance monitoring system based on 1588v2 Technology
- Leveldb source code analysis -- open the database
- Explore cloudera manager management software tuning (1)
- Using consistent hash algorithm in Presto to enhance the data cache locality of dynamic clusters
猜你喜欢
About swagger
Issue 39: MySQL time class partition write SQL considerations

How to decompile APK files

NVM download, installation and use

The 'ng' entry cannot be recognized as the name of a cmdlet, function, script file, or runnable program. Check the spelling of the name. If you include a path, make sure the path is correct, and then

Constantly changing the emergency dialing of harmonyos ETS during the new year

Error reported after NPM I

Why do you develop middleware when you are young? "You can choose your own way"
SQL basic tutorial (learning notes)

LC 300. Longest increasing subsequence
随机推荐
Coding enhances security vulnerability scanning capability and helps the team "move left safely"
Dunhuang Research Institute and Tencent have launched a new strategic cooperation to take you around the digital new silk road with AI
FPGA systematic learning notes serialization_ Day10 [sequential logic, competitive adventure, synchronous reset, asynchronous reset]
Go language GC implementation principle and source code analysis
A set of IM architecture technology dry goods for 100 million users (Part 2): reliability, orderliness, weak network optimization, etc
Easycvr, an urban intelligent video monitoring image analysis platform, plays national standard equipment videos and captures unstable packets for troubleshooting
[log service CLS] Tencent cloud game battle engine mgobe accesses CLS
Kubernetes 1.20.5 helm installation Jenkins
Leveldb source code analysis -- version management
Noi Mathematics: solution of quadratic congruence equation
Leetcode topic [array] -46- full arrangement
03. Tencent cloud IOT device side learning -- overview of mqtt control package
Failure analysis | database failure MHA is not switched
Install MySQL using Yum for Linux
Users of the Tiktok open platform are authorized to obtain the user's fan statistics and short video data
Realize business development on behalf of small programs, and 99% restore the function of service category management in the background of official account
Five steps to effectively monitor network traffic
电子元器件行业B2B电商市场模式、交易能力数字化趋势分析
Tencent cloud layer 7 load balancing log analysis and monitoring
C language | logical operators