Kubernetes migration of persistent volumes across storageclasses Complete Guide

Hello everyone , I am michelangelyang .

KubeSphere 3.3.0 （ If nothing else ~） This week GA 了 , As a KubeSphere Fangirl/fanboy , I can't wait to install RC edition Have a taste , The operation is as fierce as a tiger to open all components , After the installation, I found it a little awkward ： I used the wrong persistent storage .

my K8s There are two storage classes in the cluster （StorageClass）, One is OpenEBS Local storage provided , The other is QingCloud CSI Distributed storage provided , And by default StorageClass yes OpenEBS Provided local-hostpath, therefore KubeSphere The stateful component of uses local storage to save data by default .

Error error , I originally intended to use distributed storage as the default storage , But I forgot to csi-qingcloud Set as default StorageClass 了 , Anyway , I made a mistake in such a muddle . Although reloading can solve 99% The problem of , But as a mature YAML The engineer , It is impossible to reassemble , This problem must be solved without reloading , To reflect my temperament ！

In fact, I'm not the only one who has experienced this kind of situation , Many people will be confused to install a complete set of products and find StorageClass Wrong use. , I'm afraid it's not so easy to change back at this time . What a coincidence! This is not , This article is to help you solve this problem .

Ideas

Let's think about it first StorageClass What a few things need to be done . First, you need to reduce the number of copies applied to 0, Then create a new PVC, The old PV Data is copied to the new PV, Then let the app use the new PV, And expand the number of copies to the original number , Finally, the old PV Delete . In the whole process, it is necessary to prevent deletion PVC when Kubernetes take PV Also deleted .

Of course , There are some CSI The driver or storage backend may have more convenient data migration techniques , But this article provides a more general solution , Whatever the back-end storage is .

KubeSphere 3.3.0 Persistent volume declaration used after all components are opened （PVC） as follows ：

This article is based on Elasticsearch For example , Demonstrate how to make Elasticsearch The storage of is replaced by distributed storage from local storage .

Backup PVC and PV

The first step is to back up PVC and PV, In case the following operation fails , There is still room for repentance .

$ kubectl -n kubesphere-logging-system get pvc
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
data-elasticsearch-logging-data-0        Bound    pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9   20Gi       RWO            local-hostpath   28h
data-elasticsearch-logging-data-1        Bound    pvc-0851350a-270e-4d4d-af8d-081132c1775b   20Gi       RWO            local-hostpath   28h
data-elasticsearch-logging-discovery-0   Bound    pvc-8f32fc97-3d6e-471a-8121-655991d945a8   4Gi        RWO            local-hostpath   28h

$ kubectl -n kubesphere-logging-system get pv pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9 -o yaml > pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9.yaml
$ kubectl -n kubesphere-logging-system get pv pvc-0851350a-270e-4d4d-af8d-081132c1775b -o yaml > pvc-0851350a-270e-4d4d-af8d-081132c1775b.yaml

$ kubectl -n kubesphere-logging-system get pvc data-elasticsearch-logging-data-0 -o yaml > data-elasticsearch-logging-data-0.yaml
$ kubectl -n kubesphere-logging-system get pvc data-elasticsearch-logging-data-1 -o yaml > data-elasticsearch-logging-data-1.yaml

Copy the data

No matter PV Of accessModes yes ReadWriteOnce still ReadWriteMany, Before replicating data, reduce the number of copies applied to 0, because ReadWriteOne Only one mode can be mounted at the same time Pod, new Pod Can't mount , and ReadWriteMany If the mode does not reduce the number of copies to 0, New data may be written when copying data . So anyway , Reduce the number of copies to 0 .

$ kubectl -n kubesphere-logging-system get sts
NAME                              READY   AGE
elasticsearch-logging-data        2/2     28h
elasticsearch-logging-discovery   1/1     28h

$ kubectl -n kubesphere-logging-system scale sts elasticsearch-logging-data --replicas=0

$ kubectl -n kubesphere-logging-system get sts
NAME                              READY   AGE
elasticsearch-logging-data        0/0     28h
elasticsearch-logging-discovery   1/1     28h

Create a new PVC It's called new-data-elasticsearch-logging-data-0, Capacity and data-elasticsearch-logging-data-0 equally , And will storageClassName Specify as new StorageClass.

Create a Deployment, Will be new PV And the old PV All of them are attached , Then the old PV Copy your data to a new location PV.

stay 『 The workload 』 Interface Click 『 establish 』, The following YAML Just paste it in .

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kubesphere-logging-system
  labels:
    app: datacopy
  name: datacopy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: datacopy
  template:
    metadata:
      labels:
        app: datacopy
    spec:
      containers:
        - name: datacopy
          image: ubuntu
          command:
            - 'sleep'
          args:
            - infinity
          volumeMounts:
            - name: old-pv
              readOnly: false
              mountPath: /mnt/old
            - name: new-pv
              readOnly: false
              mountPath: /mnt/new
      volumes:
        - name: old-pv
          persistentVolumeClaim:
            claimName: data-elasticsearch-logging-data-0
        - name: new-pv
          persistentVolumeClaim:
            claimName: new-data-elasticsearch-logging-data-0

This Deployment Will be new PV And the old PV It's all mounted , Later we will take the old PV Copy your data to a new location PV.

Pod After successful startup , Click the container terminal icon to enter the container terminal .

Verify the old... In the container first PV Whether the mount point of contains application data , new PV Whether the mount point of is empty , Then execute the order (cd /mnt/old; tar -cf - .) | (cd /mnt/new; tar -xpf -), To ensure that all data ownership and permissions are inherited .

After execution , Verify new PV Whether the mount point of contains old PV The data of , And whether the ownership and authority are correctly inherited .

The task of copying data here is completed , Now we need to put datacopy The number of copies of is reduced to 0.

transfer PVC

The ideal state for migrating storage is to use old PVC, And point it to the new PV, Such a workload YAML The configuration list does not need to be changed . but PVC and PV The binding relationship between is unchangeable , To unbind them , The old must be deleted first PVC, Then create a new one with the same name PVC, And put the old PV Bind to it .

It should be noted that , By default PV Our recycling strategy is Delete, Once deleted PVC, Bound to it PV and PV The data in the will be deleted . This is what we don't want to see , So we need to modify the recycling policy , To delete PVC when PV Can be preserved .

In fact, you can adopt StorageClass To set the global recycling policy （reclaimPolicy）, If not set , The default is Delete. By command kubectl describe pv <pv-name> Check it out. PV Recycling strategy for （reclaimPolicy）：

$ kubectl describe pv pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9
Name:              pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9
Labels:            openebs.io/cas-type=local-hostpath
Annotations:       pv.kubernetes.io/provisioned-by: openebs.io/local
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-hostpath
Status:            Bound
Claim:             kubesphere-logging-system/data-elasticsearch-logging-data-0
Reclaim Policy:    Delete
...

$ kubectl describe pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Name:              pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: disk.csi.qingcloud.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/disk-csi-qingcloud-com]
StorageClass:      csi-qingcloud
Status:            Bound
Claim:             kubesphere-logging-system/new-data-elasticsearch-logging-data-0
Reclaim Policy:    Delete
...

We can go through patch The command will be old and new PV The recovery policy for is set to Retain.

$ kubectl patch pv pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9 patched

$ kubectl patch pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9 patched

️ Be careful ： The order is right PV No impact on the stability and availability of , It can be executed at any time .

Now you can replace the old with the new PVC Delete all ,PV Will not be affected in any way .

$ kubectl -n kubesphere-logging-system delete pvc data-elasticsearch-logging-data-0 new-data-elasticsearch-logging-data-0
persistentvolumeclaim "data-elasticsearch-logging-data-0" deleted
persistentvolumeclaim "new-data-elasticsearch-logging-data-0" deleted

In creating the final PVC Before , We have to make sure that the newly created PVC Can be bound to new PV On . You can see the new PV It is currently in release state , Cannot be new PVC binding ：

$ kubectl describe pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Name:              pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: disk.csi.qingcloud.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/disk-csi-qingcloud-com]
StorageClass:      csi-qingcloud
Status:            Released
Claim:             kubesphere-logging-system/new-data-elasticsearch-logging-data-0
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          20Gi
...

This is because PV stay spec.claimRef It still refers to the deleted PVC：

$ kubectl get pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  ...
  name: pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
  ...
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: new-data-elasticsearch-logging-data-0
    namespace: kubesphere-logging-system
    resourceVersion: "657019"
    uid: f4e96f69-b3be-4afe-bb52-1e8e728ca55e
  ...
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-qingcloud
  volumeMode: Filesystem

To solve this problem , You can directly command kubectl edit pv <pv-name> edit PV, take claimRef Delete all the contents of . Then look at PV Is already available （Available）：

$ kubectl describe pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Name:              pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: disk.csi.qingcloud.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/disk-csi-qingcloud-com]
StorageClass:      csi-qingcloud
Status:            Available
Claim:
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          20Gi

Eventually we need to create something with the old PVC A new with the same name PVC, And as far as possible to ensure that with the old PVC Same parameters for ：

• new PVC Name and old PVC The same name ;
• spec.volumeName Point to the new PV;
• new PVC Of metadata.annotations and metadata.labels And the old PVC Save same , Because these values may affect application deployment （ such as Helm chart etc. ）.

Final PVC The contents are as follows ：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: elasticsearch
    component: data
    release: elasticsearch-logging
    role: data
  name: data-elasticsearch-logging-data-0
  namespace: kubesphere-logging-system
spec:
  storageClassName: csi-qingcloud
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  volumeMode: Filesystem
  volumeName: pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e

stay 『 Storage volume declaration 』 Page hits 『 establish 』：

choice 『 edit YAML』, The above YAML Copy and paste the content , And then click 『 establish 』：

Finally, you can see new PVC and PV All of them are Bound state ：

$ kubectl -n kubesphere-logging-system get pvc data-elasticsearch-logging-data-0
NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
data-elasticsearch-logging-data-0   Bound    pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            csi-qingcloud   64s

$ kubectl get pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                         STORAGECLASS    REASON   AGE
pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            Retain           Bound    kubesphere-logging-system/data-elasticsearch-logging-data-0   csi-qingcloud            11h

Once again

up to now , We only moved data-elasticsearch-logging-data-0 The data of , about data-elasticsearch-logging-data-1, Just repeat the above steps , Remember to datacopy Medium PVC Change it to data-elasticsearch-logging-data-1 and new-data-elasticsearch-logging-data-0, The configuration contents in other places should also be modified to new .

Restore workload

Now all the storage has been migrated ,PVC The name remains unchanged ,PV Using new storage .

$ kubectl get pv -A|grep elasticsearch-logging-data
pvc-0851350a-270e-4d4d-af8d-081132c1775b   20Gi       RWO            Retain           Released   kubesphere-logging-system/data-elasticsearch-logging-data-1        local-hostpath            40h
pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9   20Gi       RWO            Retain           Released   kubesphere-logging-system/data-elasticsearch-logging-data-0        local-hostpath            40h
pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563   20Gi       RWO            Retain           Bound      kubesphere-logging-system/data-elasticsearch-logging-data-1        csi-qingcloud             9m53s
pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            Retain           Bound      kubesphere-logging-system/data-elasticsearch-logging-data-0        csi-qingcloud             11h

$ kubectl -n kubesphere-logging-system get pvc|grep elasticsearch-logging-data
data-elasticsearch-logging-data-0        Bound    pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            csi-qingcloud    27m
data-elasticsearch-logging-data-1        Bound    pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563   20Gi       RWO            csi-qingcloud    3m49s

Restore the number of copies of the workload to the previous number ：

$ kubectl -n kubesphere-logging-system scale sts elasticsearch-logging-data --replicas=2
statefulset.apps/elasticsearch-logging-data scaled

$ kubectl -n kubesphere-logging-system get pod -l app=elasticsearch,component=data
NAME                           READY   STATUS    RESTARTS   AGE
elasticsearch-logging-data-0   1/1     Running   0          4m12s
elasticsearch-logging-data-1   1/1     Running   0          3m42s

perfect ！

Finally, there is a little finishing work , We need to put all the new PV The recycling policy of is reset to Delete：

$ kubectl patch pv pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563 -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
persistentvolume/pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563 patched

$ kubectl patch pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
persistentvolume/pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e patched

$ kubectl get pv -A|grep elasticsearch-logging-data
pvc-0851350a-270e-4d4d-af8d-081132c1775b   20Gi       RWO            Retain           Released   kubesphere-logging-system/data-elasticsearch-logging-data-1        local-hostpath            40h
pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9   20Gi       RWO            Retain           Released   kubesphere-logging-system/data-elasticsearch-logging-data-0        local-hostpath            40h
pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563   20Gi       RWO            Delete           Bound      kubesphere-logging-system/data-elasticsearch-logging-data-1        csi-qingcloud             15m
pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            Delete           Bound      kubesphere-logging-system/data-elasticsearch-logging-data-0        csi-qingcloud             11h

Last, last , You can put the old PV All deleted ：

$ kubectl delete pv pvc-0851350a-270e-4d4d-af8d-081132c1775b
persistentvolume "pvc-0851350a-270e-4d4d-af8d-081132c1775b" deleted

$ kubectl delete pv pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9
persistentvolume "pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9" deleted

A simpler solution

Although the above solution perfectly solves the problem , But the steps are cumbersome , Is there a simpler way ？

You can try Qingyun's Cloud native backup disaster recovery SaaS service , No deployment required 、 Maintain local backup infrastructure , You can easily complete the free migration of data in a multi cloud heterogeneous environment , So as to realize multiple 、 On demand data protection and high availability of applications . And the price is close to the people , The dialogue is friendly to the whoring party , Provides 100GB Free storage for , Migrate several PV It's enough .

It's very simple to use , First Registered account , Then import Kubernetes colony . If you choose to connect through proxy Kubernetes colony , It is necessary to execute the commands in the red box in Kubernetes Install the agent in the cluster .

Then create a new managed warehouse .

Next, create a backup plan directly , choice Direct copy .

After the backup is successful , Put... In the cluster PVC and PV Delete , And reduce the number of copies of the workload to 0. Finally, create a recovery plan , Note that the source storage type name is local-hostpath The target storage type name of is set to the storage you want to migrate , After this recovery PV What is used is new StorageClass.

Finished .

summary

This paper introduces how to put Kubernetes Existing in cluster PV The data is migrated to the new PV, And create a with the same name PVC To point to the new PV, This completes the data migration of the application without any change to the configuration list of the application . Finally, it introduces how to use cloud native backup for disaster recovery SaaS Services to simplify the migration process .

This article by the blog one article many sends the platform OpenWrite Release ！