Using a Raspberry Pi to synchronize data
Backing Up with Rsync
Rsync is an excellent tool for backing up data. A Rasp Pi that is serving as a hub is naturally limited for performing backups. The limits will depend on the storage space that has been connected, the amount of data that has to be secured, and the consequent run times and storage size needed. However, a hub saves booting up a big file server if you are making quick backups of important configuration files in the /home
and /etc
directories. The file server then retrieves the saved information from the hub as soon it is back online.
You can set automatic backups with Rsnapshot [11], including hierarchical daily, weekly, or monthly generations. The software is available in the Raspbian repository and uses Rsync as a foundation, using the --link-dest
option to replace files on the target directory with hard links of identical files of the previous generation. This way, the amount of data that is actually copied is reduced in size, and several generations of backups are created and coexist without a corresponding increase in the size of the disk space necessary to store them.
One disadvantage of Rsnapshot is that, although it retrieves files from remote computers, it can only store them locally. This means it would have to run on the hub and communicate with all connected devices before starting the backup. As a result, two small bash scripts replace this functionality. You should copy the first script (Listing 5) to the hub and the second (Listing 6) locally to the computer. Listing 6 calls Listing 5 (see lines 8, 11, and 17) remotely. In this way, it prepares the backup before Rsync comes into play (line 14 in Listing 6). The Rsync command then creates a mirror of the home directory in the target directory.
Listing 5
Creating the Backup
01 #!/bin/bash 02 03 # remote backup support-script 04 05 # rotate: renames backups: name.03->name.04, name.02->name.03 <etc.> 06 # prepare: creates new backup directory name.00 07 # cleanup: removes all backup directories whose numbers exceed a limit 08 09 # configuration 10 11 backupRoot="/data/backups" 12 13 # --- rotate backups -------- 14 15 rotate() { 16 remoteHost="$1" 17 backupType="$2" 18 19 backupDir="$backupRoot/$remoteHost" 20 21 find "$backupDir/$backupType."* -type d -maxdepth 0 | sort -r | while read dir; do 22 nr="${dir##*.}" 23 nr="${nr#0}" 24 let nr1=nr+1 25 nr1=`printf "%02d" $nr1` 26 mv "$dir" "$backupDir/$backupType.$nr1" 27 done 28 } 29 30 # --- prepare backup ------- 31 32 prepare() { 33 remoteHost="$1" 34 backupType="$2" 35 backupDir="$backupRoot/$remoteHost" 36 targetDir="$backupDir/$backupType.00" 37 38 mkdir -p "$targetDir" 39 echo "$targetDir" 40 exit 0 41 } 42 43 # --- delete old backup ------ 44 45 cleanup() { 46 remoteHost="$1" 47 backupType="$2" 48 maxGen="$3" 49 backupDir="$backupRoot/$remoteHost" 50 51 while [ $maxGen -lt 100 ]; do 52 nr=`printf "%02d" $maxGen` 53 if [ ! -d "$backupDir/$backupType.$nr" ]; then 54 exit 0 55 else 56 rm -fr "$backupDir/$backupType.$nr" 57 let maxGen=maxGen+1 58 fi 59 done 60 } 61 62 # --- main program ------- 63 64 func="$1" 65 # check allowed function 66 grep -qw "$func" <<< "prepare rotate cleanup" || exit 3 67 shift 68 eval $func "$@"
Listing 6
Preparing the Backup
01 #!/bin/bash 02 03 HUB="rpi" # Host name of hub 04 rscript="/usr/local/bin/backup-server" # Script on hub 05 rota="${1:-daily}" # First argument or daily 06 07 # rotate old backup 08 ssh "$HUB" "$rscript" "rotate" "$HOSTNAME" "$rota" 09 10 # establish new backup directory and save in dir 11 dir=`ssh "$HUB" "$rscript" "prepare" "$HOSTNAME" "$rota"` 12 13 # save home directory on hub in dir 14 rsync -avz --delete --numeric-ids --relative --delete-excluded --link-dest="${dir/.00/.01}/" "$HOME" "$HUB:$dir" 15 16 # clean up superfluous generations 17 ssh "$HUB" "$rscript" "cleanup" "$HOSTNAME" "$rota" "30"
To avoid having to provide a password every time you call SSH and Rsync, you should create a key pair and copy your public key on the Rasp Pi [12]. All the listings for this article are online [13], with a modified Raspbian image.
Comfortable Image
You will find a preconfigured Raspbian image that works for the sync server introduced in this article in the /RPG/sync/
directory on the Raspberry Pi Geek magazine anonymous FTP server [13]. It uses Lighttpd as a web server and SQLite3 as a database. The Firefox sync minimal server FSyncMS, Baikal, the CalDAV/CardDAV synchronizer, and scripts for a Rsnapshot-based backup have been preinstalled.
To store the image on an SD card with a minimum capacity of 2GB, decompress the archive raspi-sync.img.zip
and copy the image onto the SD card using the command:
# dd if=raspi-sync.img of=/dev/<xxx>
Here you should replace <xxx>
with the device indicator of the SD card, which in most cases will be sdb
. Windows users should use Win32DiskImager, which you will find on the magazine DVD, instead of dd
.
The image that has been configured for operating in headless mode retrieves its IP address per DHCP. You should log in to the system via SSH with the username pi and the password raspberry. Further information, such as the predefined username and passwords for Baikal, can be found in /etc/motd
. At login, you will automatically be notified of this.
You will have to make some changes in the configuration files after the first start, because Firefox Sync and CalDavZAP both need to know the hostname on which the service runs.
Infos
- ownCloud: https://owncloud.org
- Seafile: http://seafile.com
- Seafile on the Raspberry Pi: http://draptik.github.io/blog/2014/04/21/installing-seafile-on-raspberry-pi/
- FSyncMS: https://github.com/balu-/FSyncMS
- Firefox Sync: https://blog.mozilla.org/services/2014/04/30/firefox-syncs-new-security-model/
- Baikal: http://baikal-server.com
- CalDavZAP: http://www.inf-it.com/open-source/clients/caldavzap/
- CardDavMate: http://www.inf-it.com/open-source/clients/carddavmate/
- F-Droid: https://f-droid.org
- Unison: http://www.cis.upenn.edu/~bcpierce/unison/
- Rsnapshot: http://www.rsnapshot.org
- "Remote Access" by Ruth Suehle, Raspberry Pi Handbook, 2014, 2nd edition, pg. 66, http://www.linuxpromagazine.com/Resources/Special-Editions
- Code: ftp://ftp.raspberry-pi-geek.com/pub/listings/raspberry-pi-geek.com/06
« Previous 1 2 3 4 5 Next »
Buy this article as PDF
Pages: 8
(incl. VAT)