Explain kubernetes backup and recovery tools velero | learn more about carina series phase III

Preface

Carina

  It is a cloud native local storage project led and initiated by Boyun （GitHub  The address is ：https://github.com/carina-io/carina）, At present, it has entered  CNCF  Panorama .

Carina

  It can provide services for stateful applications in the cloud native environment

High performance

、

No operation and maintenance

Of

Local storage solutions

, With storage volume lifecycle management 、LVM/RAW Panel supply 、 Intelligent scheduling 、RAID management 、 Automatic layering and other capabilities , Designed to provide very low latency for cloud native stateful Services 、 No operation and maintenance 、 Understand database data storage system .Carina  As one of the components of Bo cloud container cloud platform , It has operated stably in the production environment of multiple financial institutions for many years .

There are two traditional data backup schemes ,  One is to use the server storing data to realize snapshot based backup , The other is to deploy a proprietary backup on each target server  agent  And specify the backup data directory , Periodically copy data to external storage . The backup mechanisms of these two methods are relatively fixed , In the age of cloud primordial, it cannot adapt to the elasticity after containerization 、 Deployment scenarios such as pooling .

Cloud native storage plug-ins  Carina  For example , In data sensitive scenarios such as databases, each database cluster includes multiple computing instances , The instance may drift arbitrarily in the cluster and realize automatic fault recovery . The traditional data backup method can rapidly expand and shrink the capacity of the database cluster 、 In scenarios such as cross node drift, it is unable to automatically follow the migration of computing instances, resulting in data backup failure , So a fit  k8s  Backup tools for container scenarios are very important .

Kubernetes Backup and recovery tools ：velero

Velero  It is a disaster recovery and migration tool in the cloud native era , use  Go  Language writing , And in  github  Open source on , The open source address is ：https://github.com/vmware-tanzu/velero.Velero  From Spanish , It means sailing boat , Very much in  Kubernetes  The naming style of the community .

utilize  velero  Users can back up safely 、 Recovery and migration  Kubernetes  Cluster resources and persistent volumes . Its basic principle is to cluster data , For example, cluster resources and persistent data volumes are backed up to the object store , Pull data from the object store during recovery . In addition to disaster recovery, it can also do resource transfer , Support the migration of container applications from one cluster to another , This is also  velero  A very successful use case .

Velero  It mainly includes connecting core components , They are server and client respectively . The server runs on a specific  Kubernetes  In the cluster , The client is a command line tool that runs locally , As long as the configuration is good  kubectl  And  kubeconfig  You can use , It's simple .

Velero  Based on its implementation  kubernetes  Resource backup capability , Can be easily implemented  Kubernetes  Data backup and recovery of cluster 、 Copy  kubernetes  Cluster resources to other  kubernetes  Cluster or quickly copy the production environment to the test environment .

In terms of resource backup ,velero  Support data backup to numerous cloud storage , for example AWS S3 or S3 Compatible storage systems 、Azure Blob、Google Cloud Storage 、Aliyun OSS etc. . And back up the entire  kubernetes  The data storage engine of  etcd  comparison ,velero  The control of is more detailed , It can be done to  Kubernetes  Backup at the object level within the cluster , It can also be done by  Type、Namespace、Label  Such as object classification backup or recovery .

Velero Workflow

Take the core data backup as an example , When executed velero backup create my-backup when ：

• Velero  The client first calls  Kubernetes API  Server to create  Backup  object ;

• BackupController  Will receive notification of new  Backup  Objects are created and validated ;

• BackupController  Start the backup process , It's through a query  API  Server to obtain resources to collect data for backup ;

• BackupController  The object storage service... Will be called , for example ,AWS S3 - Upload backup file . By default ,velero backup create Supports disk snapshots of any persistent volume , You can adjust the snapshot by specifying other flags , function velero backup create --help You can view the available flags , You can also use --snapshot-volumes=false Option to disable snapshots .

About backup storage locations and volume snapshots ,Velero  There are two custom resources  BackupStorageLocation  and  VolumeSnapshotLocation, Used for configuration  Velero  Storage location of the backup and its associated persistent volume snapshots .

• BackupStorageLocation  The main back-end storage support is  S3  Compatible storage , Store all Velero The prefix in the data store and a set of other provider specific fields . such as ：Minio  And Alibaba cloud  OSS  etc.  ;

• VolumeSnapshotLocation（pv  data ）, Mainly for  PV  Take a snapshot , Need plug-ins from cloud providers , Specific fields provided entirely by the provider （ for example AWS Area ,Azure Resource group ,Portworx Snapshot type, etc ） Definition . Take the database and middleware that are most sensitive to data consistency as an example , Open source storage plug-ins  Carina  Database aware  velero  Volume snapshot function , It can realize fast backup and recovery of middleware data .

Velero Installation and use

install velero client

$ wget https://mirror.ghproxy.com/https://github.com/vmware-tanzu/velero/releases/download/v1.6.3/velero-v1.6.3-darwin-amd64.tar.gz 
$ tar -zxvf velero-v1.6.3-darwin-amd64.tar.gz && cd velero-v1.6.3-darwin-amd64 
$ mv velero /usr/local/bin && chmod +x /usr/local/bin/velero 
$ velero version


install minio Back end for data backup

Minio install Yaml The documents are as follows ：

apiVersion: v1 
kind: Namespace 
metadata: 
 name: velero 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
 namespace: velero 
 name: minio 
 labels: 
 component: minio 
spec: 
 strategy: 
 type: Recreate 
 selector: 
 matchLabels: 
 component: minio 
 template: 
 metadata: 
 labels: 
 component: minio 
 spec: 
 volumes: 
 - name: storage 
 emptyDir: {} 
 - name: config 
 emptyDir: {} 
 containers: 
 - name: minio 
 image: minio/minio:latest 
 imagePullPolicy: IfNotPresent 
 args: 
 - server 
 - /storage 
 - --config-dir=/config 
 - --console-address=:9001 
 env: 
 - name: MINIO_ACCESS_KEY 
 value: "minio" 
 - name: MINIO_SECRET_KEY 
 value: "minio123" 
 ports: 
 - containerPort: 9000 
 - containerPort: 9001 
 volumeMounts: 
 - name: storage 
 mountPath: "/storage" 
 - name: config 
 mountPath: "/config" 
--- 
apiVersion: v1 
kind: Service 
metadata: 
 namespace: velero 
 name: minio 
 labels: 
 component: minio 
spec: 
 type: NodePort 
 ports: 
 - name: api 
 port: 9000 
 targetPort: 9000 
 - name: console 
 port: 9001 
 targetPort: 9001 
 selector: 
 component: minio 
--- 
apiVersion: batch/v1 
kind: Job 
metadata: 
 namespace: velero 
 name: minio-setup 
 labels: 
 component: minio 
spec: 
 template: 
 metadata: 
 name: minio-setup 
 spec: 
 restartPolicy: OnFailure 
 volumes: 
 - name: config 
 emptyDir: {} 
 containers: 
 - name: mc 
 image: minio/mc:latest 
 imagePullPolicy: IfNotPresent 
 command: 
 - /bin/sh 
 - -c 
 - "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero" 
 volumeMounts: 
 - name: config 
 mountPath: "/config" 


install Mini, And check the resource creation .

$ kubectl apply -f ./00-minio-deployment.yaml 
$ kubectl get pods -n velero 
NAME READY STATUS RESTARTS AGE
minio-58dc5cf789-z2777 0/1 ContainerCreating 0 14s
minio-setup-dz4jb 0/1 ContainerCreating 0 6s
$ kubectl get svc -n velero 
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
minio NodePort 10.96.13.35 <none> 9000:30693/TCP,9001:32351/TCP 17s


After the service has been started , You can log in.  minio  see  velero/velero  Of  bucket  Whether to create successfully .

install  velero  Server side  , Use s3  As the storage

• establish  minio  voucher

$ cat > credentials-velero <<EOF
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123
EOF
#  install velero
$ cp velero /usr/bin/
#  Enable fast completion
$ velero completion bash


• Use the official  restic  Component backup  pv

$ velero install \
 --image velero/velero:v1.6.3 \
 --plugins velero/velero-plugin-for-aws:v1.0.0 \
 --provider aws \
 --bucket velero \
 --namespace velero \
 --secret-file ./credentials-velero \
 --velero-pod-cpu-request 200m \
 --velero-pod-mem-request 200Mi \
 --velero-pod-cpu-limit 1000m \
 --velero-pod-mem-limit 1000Mi \
 --use-volume-snapshots=false \
 --use-restic \
 --restic-pod-cpu-request 200m \
 --restic-pod-mem-request 200Mi \
 --restic-pod-cpu-limit 1000m \
 --restic-pod-mem-limit 1000Mi \
 --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000


among , Several important parameters and their descriptions are as follows ：

--provider： The statement uses  Velero  Plug in type .
--plugins： Use  S3 API  Compatible plug-ins  “velero-plugin-for-aws ”.
--bucket： Tencent's cloud  COS  Created bucket name .
--secret-file： visit  COS  Access credentials file for , See created above  “credentials-velero” Voucher file .
--use-restic： Use open source free backup tools  restic  Backup and restore persistent volume data .
--default-volumes-to-restic： Use  restic  To back up everything Pod volume , The premise is that you need to turn on  --use-restic  Parameters .
--backup-location-config： Backup bucket access related configuration .
--region： compatible  S3 API  Of  COS  The bucket area , For example, if the founding area is Guangzhou ,region  Parameter values for “ap-guangzhou”.
--s3ForcePathStyle： Use  S3  File path format .
--s3Url：COS  Compatible  S3 API  Access address
--use-volume-snapshots=false  To turn off the snapshot backup of storage volume data .


After the installation command is executed , wait for  Velero  and  restic  When the workload is ready , Check if the configured storage location is available .

$ velero backup-location get 

apiVersion: velero.io/v1 
kind: BackupStorageLocation 
metadata: 
 name: default 
 namespace: velero 
spec: 
#  Only  aws gcp azure 
 provider: aws 
 objectStorage: 
 bucket: myBucket 
 prefix: backup 
 config: 
 region: us-west-2 
 profile: "default" 
 s3ForcePathStyle: "false" 
 s3Url: http://minio:9000



thus  velero  It's all deployed .

velero Function is introduced

Create a backup

velero  Backup of all objects is supported , Or by type , Namespace and / Or label filtering objects

$ velero create backup $NAME [flags]
$ velero backup create pvc-backup-1 --snapshot-volumes --include-namespaces nginx-example --default-volumes-to-restic --volume-snapshot-locations default


among ：

--include-namespaces: Back up all resources under the namespace , Cluster resources are not included

--include-resources: Type of resource to back up

--include-cluster-resources: Whether to back up cluster resources   This option can have three possible values ： true： Include all cluster wide resources ; false： Cluster wide resources are not included ; nil （“ Automatically ” Or not ）

--selector: Select the matching resource backup through the tag

--exclude-namespaces: Resources under this namespace will not be backed up during backup

--exclude-resources: This type of resource will not be backed up during backup

http://--velero.io/exclude-from-backup=true: When the tag selector matches the resource , If the resource has this tag , No backup

meanwhile , You can also use  –ordered-resources  Parameters , Back up specific kinds of resources in a specific order , You need to specify the resource name and the object name list of the resource , Resource object names are separated by commas , Its name format is “ Namespace / Resource name ”, For cluster wide resources , Just use the resource name . Key value pairs in the map are separated by semicolons , Resource types are plural .

$ velero backup create backupName --include-cluster-resources=true --ordered-resources 'pods=ns1/pod1,ns1/pod2;persistentvolumes=pv4,pv8' --include-namespaces=ns1

$ velero backup create backupName --ordered-resources 'statefulsets=ns1/sts1,ns1/sts0' --include-namespaces=n


Scheduled backup ：

$ velero schedule create <SCHEDULE NAME> --schedule &quot;0 7 * * *&quot;

$ velero create schedule NAME --schedule=&quot;@every 6h&quot;

$ velero create schedule NAME --schedule=&quot;@every 24h&quot; --include-namespaces web

$ velero create schedule NAME --schedule=&quot;@every 168h&quot; --ttl 2160h0m0s


Examples of backup advanced usage

• In a single Velero Create snapshots of more than one persistent volume in the backup

$ velero snapshot-location create ebs-us-east-1 \
 --provider aws \
 --config region=us-east-1

$ velero snapshot-location create portworx-cloud \
 --provider portworx \
 --config type=cloud

$ velero backup create full-cluster-backup \
 --volume-snapshot-locations ebs-us-east-1,portworx-cloud 


• Store backups in different object buckets in different regions

$ velero backup-location create default \
 --provider aws \
 --bucket velero-backups \
 --config region=us-east-1

$ velero backup-location create s3-alt-region \
 --provider aws \
 --bucket velero-backups-alt \
 --config region=us-west-1

$ velero backup create full-cluster-alternate-location-backup \
 --storage-location s3-alt-region


• For storage volumes provided by the public cloud , Store some snapshots locally , Some are stored in the public cloud

$ velero snapshot-location create portworx-local \
 --provider portworx \
 --config type=local

$ velero snapshot-location create portworx-cloud \
 --provider portworx \
 --config type=cloud

$ velero backup create cloud-snapshot-backup \
 --volume-snapshot-locations portworx-cloud 


• Use storage location

$ velero backup-location create default \
 --provider aws \
 --bucket velero-backups \
 --config region=us-west-1

$ velero snapshot-location create ebs-us-west-1 \
 --provider aws \
 --config region=us-west-1

$ velero backup create full-cluster-backup


View backup tasks .

When the backup task status is  “Completed” , And the number of errors is  0 , Indicates that the backup task is completed without any errors , You can query through the following command ：

 $ velero backup get 


By temporarily updating the backup storage location to read-only mode , It can prevent the creation or deletion of backup objects in the backup storage location during the restore process .

$ kubectl patch backupstoragelocation default --namespace velero \
 --type merge \
 --patch '{"spec":{"accessMode":"ReadOnly"}}'
 
velero backup-location get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
default aws velero Unknown Unknown ReadWrite true


Restore backup data

$ velero restore create --from-backup <backup-name>
$ velero restore create --from-backup pvc-backup-1 --restore-volumes


View recovery tasks .

$ velero restore get 


When the restore is complete , Don't forget to restore the backup storage location to read-write mode , For the next backup task ：

$ kubectl patch backupstoragelocation default --namespace velero \
 --type merge \
 --patch '{"spec":{"accessMode":"ReadWrite"}}'


Backup  hooks  Introduce

Velero  Support to execute some preset commands in the container before and after the backup task is executed , This method is very effective for data consistency .velero  Two methods are supported to specify the hook , One is  pod  Own annotation declaration , The other is to define  Backup  During the mission  Spec  In a statement .

• Pre hooks

pre.hook.backup.velero.io/container: The container that will execute the command , The default is pod The first container in , Optional .

pre.hook.backup.velero.io/command: Commands to execute , If more than one parameter is required , Please specify this command as JSON Array . for example ：["/usr/bin/uname", "-a"]

pre.hook.backup.velero.io/on-error: How to handle if the command returns a non-zero exit code . The default is “Fail”, Valid values are “Fail” and “Continue”, Optional .

pre.hook.backup.velero.io/timeout: Time to wait for the command to execute , If the command exceeds the timeout , It is considered that the hook fails . The default is 30 second , Optional .



• Post hooks

post.hook.backup.velero.io/container: The container that will execute the command , The default is pod The first container in , Optional .

post.hook.backup.velero.io/command: Commands to execute , If more than one parameter is required , Please specify this command as JSON Array . for example ：["/usr/bin/uname", "-a"]

post.hook.backup.velero.io/on-error: How to handle if the command returns a non-zero exit code . The default is “Fail”, Valid values are “Fail” and “Continue”, Optional .

post.hook.backup.velero.io/timeout: Time to wait for the command to execute , If the command exceeds the timeout , It is considered that the hook fails . The default is 30 second , Optional


Restore  hooks  Introduce

Velero  Support restore  hooks, Custom actions that can be performed before or after the restore task . There are two forms of definition ：

• InitContainer Restore Hooks： These will be restored in  Pod  The application container for will be  init  The container is added to the restored  pod  in , To perform any necessary settings .

init.hook.restore.velero.io/container-image: To add init Container image of container

init.hook.restore.velero.io/container-name: To add init Name of the container

init.hook.restore.velero.io/command: The task or command to be executed in the initialization container


Such as before backup , Use the following command to add comments to Pod：

kubectl annotate pod -n <POD_NAMESPACE> <POD_NAME> \
 init.hook.restore.velero.io/container-name=restore-hook \
 init.hook.restore.velero.io/container-image=alpine:latest \
 init.hook.restore.velero.io/command='[&quot;/bin/ash&quot;, &quot;-c&quot;, &quot;date&quot;]'



• Exec Restore Hooks： Can be used in restored Kubernetes pod Execute custom commands or scripts in the container of .

post.hook.restore.velero.io/container:; perform hook The name of the container , The default is the first container , Optional

post.hook.restore.velero.io/command: Commands to be executed in the container , Required

post.hook.restore.velero.io/on-error: How to handle execution failure , Valid values are Fail and Continue, The default is Continue, Use Continue Pattern , Only record execution failures ; Use Fail Mode time , Will not be on their own other hook, The restored state will be PartiallyFailed, Optional

post.hook.restore.velero.io/exec-timeout: How long to wait after starting execution , The default is 30 second , Optional

post.hook.restore.velero.io/wait-timeout: Time to wait for the container to be ready , This time should be long enough , To enable the container to start , and


Such as before backup , Use the following command to add comments to Pod

kubectl annotate pod -n <POD_NAMESPACE> <POD_NAME> \
 post.hook.restore.velero.io/container=postgres \
 post.hook.restore.velero.io/command='[&quot;/bin/bash&quot;, &quot;-c&quot;, &quot;psql < /backup/backup.sql&quot;]' \
 post.hook.restore.velero.io/wait-timeout=5m \
 post.hook.restore.velero.io/exec-timeout=45s \
 post.hook.restore.velero.io/on-error=Continue


Velero Analysis of some key problems

Velero Can resources be restored to a namespace different from their backup source ？

Yes , have access to --namespace-mappings Parameter to specify ：

velero restore create RESTORE_NAME \
 --from-backup BACKUP_NAME \
 --namespace-mappings old-ns-1:new-ns-1,old-ns-2:new-ns-2


After the restore operation , Existing  NodePort  Type of  service  How to deal with it ？

Velero  There is a parameter , Allows the user to decide to keep the original  nodePorts.

velero restore create The subcommand has  --preserve-nodeports Sign protection services nodePorts. This flag is used to preserve the original... From the backup nodePorts, It can be used as --preserve-nodeports or --preserve-nodeports=true  If this flag is given , be Velero In restoring Service Will not delete nodePorts, Instead, try to use what was written during the backup nodePorts.

velero How to implement a consistent backup strategy without affecting the business , And upload the backup data to the object storage ?

If it is based on velero Achieve database consistency , Need to use velero Of hook, The database is backed up before quiesce operation , Backup complete unquiesce. For the backup itself , have access to restic Come on copy data （ But no snapshots ）, Or use snapshots .