Using a Raspberry Pi to synchronize data

Backing Up with Rsync

Rsync is an excellent tool for backing up data. A Rasp Pi that is serving as a hub is naturally limited for performing backups. The limits will depend on the storage space that has been connected, the amount of data that has to be secured, and the consequent run times and storage size needed. However, a hub saves booting up a big file server if you are making quick backups of important configuration files in the /home and /etc directories. The file server then retrieves the saved information from the hub as soon it is back online.

You can set automatic backups with Rsnapshot [11], including hierarchical daily, weekly, or monthly generations. The software is available in the Raspbian repository and uses Rsync as a foundation, using the --link-dest option to replace files on the target directory with hard links of identical files of the previous generation. This way, the amount of data that is actually copied is reduced in size, and several generations of backups are created and coexist without a corresponding increase in the size of the disk space necessary to store them.

One disadvantage of Rsnapshot is that, although it retrieves files from remote computers, it can only store them locally. This means it would have to run on the hub and communicate with all connected devices before starting the backup. As a result, two small bash scripts replace this functionality. You should copy the first script (Listing 5) to the hub and the second (Listing 6) locally to the computer. Listing 6 calls Listing 5 (see lines 8, 11, and 17) remotely. In this way, it prepares the backup before Rsync comes into play (line 14 in Listing 6). The Rsync command then creates a mirror of the home directory in the target directory.

Listing 5

Creating the Backup

01 #!/bin/bash
02
03 # remote backup support-script
04
05 # rotate: renames backups: name.03->name.04, name.02->name.03 <etc.>
06 # prepare: creates new backup directory name.00
07 # cleanup: removes all backup directories whose numbers exceed a limit
08
09 # configuration
10
11 backupRoot="/data/backups"
12
13 # --- rotate backups --------
14
15 rotate() {
16   remoteHost="$1"
17   backupType="$2"
18
19   backupDir="$backupRoot/$remoteHost"
20
21   find "$backupDir/$backupType."* -type d -maxdepth 0 | sort -r | while read dir; do
22     nr="${dir##*.}"
23     nr="${nr#0}"
24     let nr1=nr+1
25     nr1=`printf "%02d" $nr1`
26     mv "$dir" "$backupDir/$backupType.$nr1"
27   done
28 }
29
30 # --- prepare backup -------
31
32 prepare() {
33   remoteHost="$1"
34   backupType="$2"
35   backupDir="$backupRoot/$remoteHost"
36   targetDir="$backupDir/$backupType.00"
37
38   mkdir -p "$targetDir"
39   echo "$targetDir"
40   exit 0
41 }
42
43 # --- delete old backup ------
44
45 cleanup() {
46   remoteHost="$1"
47   backupType="$2"
48   maxGen="$3"
49   backupDir="$backupRoot/$remoteHost"
50
51   while [ $maxGen -lt 100 ]; do
52     nr=`printf "%02d" $maxGen`
53     if [ ! -d "$backupDir/$backupType.$nr" ]; then
54       exit 0
55     else
56       rm -fr "$backupDir/$backupType.$nr"
57       let maxGen=maxGen+1
58     fi
59   done
60 }
61
62 # --- main program -------
63
64 func="$1"
65 # check allowed function
66 grep -qw "$func" <<< "prepare rotate cleanup" || exit 3
67 shift
68 eval $func "$@"

Listing 6

Preparing the Backup

01 #!/bin/bash
02
03 HUB="rpi"                             # Host name of hub
04 rscript="/usr/local/bin/backup-server" # Script on hub
05 rota="${1:-daily}"                    # First argument or daily
06
07 # rotate old backup
08 ssh "$HUB" "$rscript" "rotate" "$HOSTNAME" "$rota"
09
10 # establish new backup directory and save in dir
11 dir=`ssh "$HUB" "$rscript" "prepare" "$HOSTNAME" "$rota"`
12
13 # save home directory on hub in dir
14 rsync -avz --delete --numeric-ids --relative --delete-excluded --link-dest="${dir/.00/.01}/" "$HOME" "$HUB:$dir"
15
16 # clean up superfluous generations
17 ssh "$HUB" "$rscript" "cleanup" "$HOSTNAME" "$rota" "30"

To avoid having to provide a password every time you call SSH and Rsync, you should create a key pair and copy your public key on the Rasp Pi [12]. All the listings for this article are online [13], with a modified Raspbian image.

Comfortable Image

You will find a preconfigured Raspbian image that works for the sync server introduced in this article in the /RPG/sync/ directory on the Raspberry Pi Geek magazine anonymous FTP server [13]. It uses Lighttpd as a web server and SQLite3 as a database. The Firefox sync minimal server FSyncMS, Baikal, the CalDAV/CardDAV synchronizer, and scripts for a Rsnapshot-based backup have been preinstalled.

To store the image on an SD card with a minimum capacity of 2GB, decompress the archive raspi-sync.img.zip and copy the image onto the SD card using the command:

# dd if=raspi-sync.img of=/dev/<xxx>

Here you should replace <xxx> with the device indicator of the SD card, which in most cases will be sdb. Windows users should use Win32DiskImager, which you will find on the magazine DVD, instead of dd.

The image that has been configured for operating in headless mode retrieves its IP address per DHCP. You should log in to the system via SSH with the username pi and the password raspberry. Further information, such as the predefined username and passwords for Baikal, can be found in /etc/motd. At login, you will automatically be notified of this.

You will have to make some changes in the configuration files after the first start, because Firefox Sync and CalDavZAP both need to know the hostname on which the service runs.

The Author

Bernhard Bablok works at Allianz Managed Operations and Services SE as an SAP HR developer. He likes to listen to music, go biking, and take long walks. He also finds time to tackle projects involving Linux and object orientation. He can be reached via mail@bablokb.de.

Buy this article as PDF

Express-Checkout as PDF

Pages: 8

Price $2.95
(incl. VAT)

Buy Raspberry Pi Geek

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content