Topolvm: kubernetes local persistence scheme based on LVM, capacity aware, dynamically create PV, and easily use local disk

High quality resource sharing

Learning route guidance （ Click unlock ）	Knowledge orientation	Crowd positioning
🧡 Python Actual wechat ordering applet 🧡	Progressive class	This course is python flask+ Perfect combination of wechat applet , From the deployment of Tencent to the launch of the project , Create a full stack ordering system .
Python Quantitative trading practice	beginner	Take you hand in hand to create an easy to expand 、 More secure 、 More efficient quantitative trading system
Read the directory

• Text
• Previous recommendation

Back to the top ## Text

R & D test scenario , The general pursuit is to quickly start the environment with one click , Horizontal dynamic replication , One person, one set , Use as you go , Go as soon as you use it . As the user , It doesn't care what the actual physical resources are , Where does the environment start , Just declare your own usage requirements . But as a solution builder and infrastructure brace , But we know , To provide a better solution , There are still many problems , And it's not easy .

For example, in the past , I was once puzzled by how to gracefully release the local physical disk for business use , Especially local HDD Data disk .

There is a background , our Kubernetes The R & D test cluster is built with the over guaranteed machines that have retired online , Then qiniu started cloud storage , So many of our machines are multi disk storage intensive machines ( Like hanging 12 block 4T Disk ). So how to make better use of these disks , It's a question .

One solution is to make these disks into network storage , And then through qiniu's own cloud service or ceph Wait for the system to provide , It's certainly possible . However, some businesses just want to use physical disks , So what should we do ？

throughout Kubernetes At present, several native solutions are provided , The author found that they are not perfect :

• Emptydir Non persistent scheme ,Pod Delete ,EmptyDir It will be emptied . in addition Emptydir It uses rootfs Storage space , This space may be on the system disk , Therefore, it should be used with caution .
• HostPath Persistence scheme , But the security risks are high , The government does not recommend . in addition , It is also inconvenient from the point of view of use . For example, as a user , You must first know the situation of the target system , Only after you know the specific drive letter and directory location can you HostPath Quote correctly in . But the reality is from the perspective of cluster security , The user may not have the right to log in to the machine directly . And even if your Pod Occupied the directory of a host computer , Nor can we rule out the second occupation by others , Or misoperation . therefore HostPath The usage scenarios for are actually very limited .
• Local Persistent Volume Persistence scheme , With PVC/PV To use local storage . Good idea. , But it cannot be created dynamically PV, about PV For providers , The mental burden is high . Of course, the community now offers Local Static Provisioner, To some extent, it simplifies the mental burden of this piece , But you still need to plan directories or partitions in advance , It's not enough .

I think , Ideal local disk usage , Should be Apply on demand , Spatial isolation , And automated lifecycle management . In this way, it is convenient for the end user to use , It can also reduce operation and maintenance support , Increase of efficiency . There are three key technical points here :

• Apply on demand It means that it is best to PVC+StorageClass To provide services , Achieve PV Dynamically create . And to achieve that , Capacity aware Is the key , Because it is obviously unreasonable to schedule to nodes with insufficient space . It's better to combine PVC Capacity requested +Node Remaining capacity on , Comprehensively select the best node to bind .
• Spatial isolation Make sure that the space requested by the user is sufficient , Free from encroachment . From here , Simply use the directory of the file system as PV But it is obviously inappropriate not to isolate each other in capacity .
• Automated lifecycle management dynamic Provisioning Is a strong demand .

Synthesize the above three points , We will find that based on LVM Or partition technology +CSI The implementation of the , When it is more in line with the above user experience , and TopoLVM It's such a project .

Address : https://github.com/topolvm/topolvm

TopoLVM Is based on LVM Of Kubernetes Localized disk solutions , With CSI The form is provided to users for use . Currently, the following functions are supported :

• dynamic Provisioning
• Support native block volumes （Raw Block Volume）
• Volume Telescopic

The overall structure is as follows :

It is worth noting that , In earlier versions , In order to dynamically perceive the remaining storage capacity on the node ,TopoLVM Designed a custom extension scheduler （ Upper figure topolvm-scheduler part ）, Convenient in Scheduling Stage is Pod Bind appropriate Node. And in the Kubernetes 1.21 after ,Kubernete Native support already Storage Capacity Tracking The ability of , The whole implementation of this piece becomes much more elegant ,topolvm-Scheduler It's no longer necessary .

Of course , To realize TopoLVM The core principle of , Besides understanding CSI Write out of specification , The most important thing is to understand LVM Related technology . And it is precisely because through LVM Can dynamically create LV, Dynamic expansion and contraction ,TopoLVM To support dynamic Provisioning Related capabilities .

however , Although as an open source project TopoLVM Basically enough , But the richness is slightly insufficient . Recently, Boyun has also opened its cloud native local disk management solution Carina, It looks more perfect .

Project address : https://github.com/carina-io/carina

Carina In addition to providing based on LVM Out of the plan , It also supports raw disk partition , as well as IOPS Limit etc. , More features . The code organization specification is also more suitable for the cloud native community , The whole is worth exploring .

Reference link

• Volume: https://kubernetes.io/docs/concepts/storage/volumes/
• PV: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
• Storage Capacity: https://kubernetes.io/docs/concepts/storage/storage-capacity/
• LVM: https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)

Back to the top ## Previous recommendation

• Talk about leadership and team leadership
• Chat Kubernetes Pod or Namespace Stuck in Terminating State scene
• Build efficiently Presubmit Checkpoint , Landing test left shift best practice
• Talk about test environment management and practice
• How we do it go System coverage collection ？
• Chat Go Code coverage technology and best practices