Raspberry pie self built NAS cloud disk -- automatic data backup

Take yours. Raspberry pie Become a safe place for data .

stay 《 Raspberry pie build by oneself NAS Cloud disk 》 Series of Chapter one In the article , We discussed establishing NAS Some basic steps of , Added two pieces 1TB Storage hard disk drive （ One for data storage , One for data backup ）, And through the network file system （NFS） Mount the data storage disk to the remote terminal . This is the second article in this series , We will discuss automatic data backup . Automatic data backup ensures the safety of data , It provides convenience for data recovery after hardware damage and reduces unnecessary trouble caused by file misoperation .

Backup policy

We have changed from small NAS Imagine a backup strategy and get started . I suggest having a time node every day 、 Plan to back up data , To prevent interference with our normal access NAS, For example, the backup time point avoids being accessed NAS And write the time point of the file . for instance , You can do it every morning 2 Click to back up data .

in addition , You also have to decide how long your daily backups need to be kept , Because if there is no time limit , The storage space will soon be used up . Generally, the daily backup can be kept for a week , If the data goes wrong , You can easily recover the original data from the backup . But what if you need to restore the data to a longer time ago ？ You can keep the backup files every Monday for one month 、 Keep the backup every month for a longer time . Let's keep the monthly backup for one year , Every year, the backup is kept longer 、 For example, five years .

such , A large number of backups will be generated on the backup disk within five years ：

• Once a week 7 Daily backup
• monthly 4 Weekly backup
• Every year, 12 Monthly backup
• Every five years 5 One year backup

You should remember , The backup disk we built is the same size as the data disk （ Every 1 TB）. How to be more than 10 individual 1TB The backup of data is stored from the data disk to only 1TB The size of the backup disk ？ If you create a full backup , It's obviously not possible . therefore , You need to create incremental backups , Each backup is created based on the data of the previous backup . Incremental backup will not double the storage space every other day , It only takes up a little more space every day .

Here is my situation ： my NAS since 2016 year 8 It will start operation in June , There is 20 Backup . at present , I stored on the data disk 406GB The file of . My backup disk is used 726GB. Of course , The utilization rate of backup disk space depends largely on the frequency of data changes , But as you can see , Incremental backup will not occupy 20 Space required for a full backup . However , as time goes on ,1TB There may not be enough space for backup . Once data growth approaches 1TB Limit （ Or any backup disk capacity ）, You should choose a larger backup disk space and move the data over .

utilize rsync Data backup

utilize rsync Command line tools can generate full backups .

1. [email protected]:~ $ rsync -a /nas/data/ /nas/backup/2018-08-01

This command will be attached to /nas/data/ The data in the data disk under the directory has been completely copied and backed up . The backup file is saved in /nas/backup/2018-08-01 Under the table of contents .-a The parameter is backup in archive mode , This will back up all metadata , For example, the revision date of the document 、 jurisdiction 、 Owner and soft link file .

Now? , You are already in 8 month 1 A complete initial backup was created on the th , You will be in 8 month 2 Create the first incremental backup every day .

1. [email protected]:~ $ rsync -a --link-dest /nas/backup/2018-08-01/ /nas/data/ /nas/backup/2018-08-02

The above line of code creates another one about /nas/data Backup of data in the directory . The backup path is /nas/backup/2018-08-02. Parameters here --link-dest Specifies the path where a backup file is located . such , This backup will work with /nas/backup/2018-08-01 Compare the backup of , Back up only the modified files , Unmodified files will not be copied , Instead, create a hard link to them in the previous backup file .

When using hard linked files in backup files , You generally don't notice the difference between hard links and the original copy . They behave exactly the same , If you delete one of the hard links or files , Others still exist . You can think of them as two different entries to the same file . Here is an example ：

The left box shows the original data status after the second backup . The square in the middle is the backup of yesterday . There were only pictures in yesterday's backup file1.jpg did not file2.txt . The box on the right reflects today's incremental backup . The incremental backup command creates a file2.txt. because file1.jpg It has not been modified since yesterday , So today I created a hard link , It doesn't take up extra space on the disk .

Automated backup

You certainly don't want to input commands for data backup every morning . You can create a task to call the following script regularly to make it backup automatically .

1. #!/bin/bash
2. TODAY=$(date+%Y-%m-%d)
3. DATADIR=/nas/data/
4. BACKUPDIR=/nas/backup/
5. SCRIPTDIR=/nas/data/backup_scripts
6. LASTDAYPATH=${BACKUPDIR}/$(ls${BACKUPDIR}|tail-n1)
7. TODAYPATH=${BACKUPDIR}/${TODAY}
8. if[[!-e ${TODAYPATH}]];then
9. mkdir-p ${TODAYPATH}
10. fi
11. rsync-a--link-dest ${LASTDAYPATH}${DATADIR}${TODAYPATH}[email protected]
12. ${SCRIPTDIR}/deleteOldBackups.sh

The first code specifies the data path 、 Backup path 、 Script path and backup path of yesterday and today . The second code calls rsync command . The last piece of code executes deleteOldBackups.sh Script , It will clear some expired unnecessary backup data . If you don't want to call frequently deleteOldBackups.sh, You can also execute it manually .

The following is a simple and complete example script of the backup strategy discussed today .

1. #!/bin/bash
2. BACKUPDIR=/nas/backup/
3. functionlistYearlyBackups(){
4. foriin012345
5. dols${BACKUPDIR}|egrep"$(date +%Y -d "${i}year ago")-[0-9]{2}-[0-9]{2}"|sort-u|head-n1
6. done
7. }
8. functionlistMonthlyBackups(){
9. foriin0123456789101112
10. dols${BACKUPDIR}|egrep"$(date +%Y-%m -d "${i}month ago")-[0-9]{2}"|sort-u|head-n1
11. done
12. }
13. functionlistWeeklyBackups(){
14. foriin01234
15. dols${BACKUPDIR}|grep"$(date +%Y-%m-%d -d "lastmonday-${i}weeks")"
16. done
17. }
18. functionlistDailyBackups(){
19. foriin0123456
20. dols${BACKUPDIR}|grep"$(date +%Y-%m-%d -d "-${i}day")"
21. done
22. }
23. functiongetAllBackups(){
24. listYearlyBackups
25. listMonthlyBackups
26. listWeeklyBackups
27. listDailyBackups
28. }
29. functionlistUniqueBackups(){
30. getAllBackups|sort-u
31. }
32. functionlistBackupsToDelete(){
33. ls${BACKUPDIR}|grep-v-e"$(echo -n $(listUniqueBackups) |sed "s/ /\\\|/g")"
34. }
35. cd${BACKUPDIR}
36. listBackupsToDelete|whileread file_to_delete;do
37. rm-rf ${file_to_delete}
38. done

This script will first list all the backup files that need to be saved according to your backup strategy , Then it will delete those backup directories that are no longer needed .

Now create a scheduled task to execute the above code . With root User rights open crontab -e, Enter the following command , It will create an early morning every day 2 Go ahead and execute /nas/data/backup_scripts/daily.sh Scheduled tasks for .

1. 02***/nas/data/backup_scripts/daily.sh

For creating scheduled tasks, please refer to cron Create a scheduled task .

• When there is no backup task , Uninstall your backup disk or mount it as a read-only disk ;
• Take advantage of remote The server As your backup disk , In this way, data can be synchronized through the Internet

You can also use the following methods to strengthen your backup strategy , To prevent accidental deletion or destruction of backup data ：

The example of backup strategy in this article is to back up some data that I think is valuable , You can also modify these strategies according to your personal needs .

I will be 《 Raspberry pie build by oneself NAS Cloud disk 》 The third article in the series discusses Nextcloud.Nextcloud Provides a more convenient way to access NAS Data on the cloud disk and it also provides offline operations , You can also synchronize your data in the client .