Duplicity + GPG Compression

Duplicity home directory with mostly binary files (pdf, images, etc) with default compression:

$ duplicity --encrypt-key=96907DB4 --sign-key=96907DB4 --include=/home/nitro/svn --exclude=/** --volsize=250 / file:///mnt/backup/duplicity/default

--------------[ Backup Statistics ]--------------
StartTime 1276259659.02 (Fri Jun 11 07:34:19 2010)
EndTime 1276259918.87 (Fri Jun 11 07:38:38 2010)
ElapsedTime 259.85 (4 minutes 19.85 seconds)
SourceFiles 34343
SourceFileSize 1758582413 (1.64 GB)
NewFiles 34343
NewFileSize 1758582413 (1.64 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 34343
RawDeltaSize 1700087247 (1.58 GB)
TotalDestinationSizeChange 1134113085 (1.06 GB)
Errors 4
-------------------------------------------------

Duplicity home directory with mostly binary files (pdf, images, etc) with bzip2:

$ duplicity --encrypt-key=96907DB4 --sign-key=96907DB4 --include=/home/nitro/svn --exclude=/** --volsize=250 --gpg-options='--compress-algo=bzip2 --bzip2-compress-level=9'  / file:///mnt/backup/duplicity/bz2

--------------[ Backup Statistics ]--------------
StartTime 1276258959.45 (Fri Jun 11 07:22:39 2010)
EndTime 1276259564.09 (Fri Jun 11 07:32:44 2010)
ElapsedTime 604.64 (10 minutes 4.64 seconds)
SourceFiles 34343
SourceFileSize 1758582413 (1.64 GB)
NewFiles 34343
NewFileSize 1758582413 (1.64 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 34343
RawDeltaSize 1700087247 (1.58 GB)
TotalDestinationSizeChange 1082959785 (1.01 GB)
Errors 4
-------------------------------------------------

 Duplicity backup of /etc with mostly plain-text files with default compression:

$ duplicity --encrypt-key=96907DB4 --sign-key=96907DB4 --include=/etc --exclude=/** --volsize=250 / file:///mnt/backup/duplicity/default

--------------[ Backup Statistics ]--------------
StartTime 1276260094.21 (Fri Jun 11 07:41:34 2010)
EndTime 1276260099.76 (Fri Jun 11 07:41:39 2010)
ElapsedTime 5.55 (5.55 seconds)
SourceFiles 1393
SourceFileSize 6574960 (6.27 MB)
NewFiles 1392
NewFileSize 6570864 (6.27 MB)
DeletedFiles 2
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1394
RawDeltaSize 5887730 (5.61 MB)
TotalDestinationSizeChange 968617 (946 KB)
Errors 31
-------------------------------------------------

Duplicity backup of /etc with mostly plain-text files with bzip2 compression:

$ duplicity --encrypt-key=96907DB4 --sign-key=96907DB4 --include=/etc --exclude=/** --volsize=250 --gpg-options='--compress-algo=bzip2 --bzip2-compress-level=9'  / file:///mnt/backup/duplicity/bz2

--------------[ Backup Statistics ]--------------
StartTime 1276260124.50 (Fri Jun 11 07:42:04 2010)
EndTime 1276260127.12 (Fri Jun 11 07:42:07 2010)
ElapsedTime 2.62 (2.62 seconds)
SourceFiles 1393
SourceFileSize 6574960 (6.27 MB)
NewFiles 1393
NewFileSize 6574960 (6.27 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1393
RawDeltaSize 5887730 (5.61 MB)
TotalDestinationSizeChange 845953 (826 KB)
Errors 31
-------------------------------------------------

In conclusion, it seems that the default options are sufficient in most cases for my backups and the time trade-off isn't worth using bzip2.

DreamHost and Duplicity Backups

I started on Monday backing up my home computers to DreamHost backup using duplicity.

DreamHost gives their shared hosting users 50GB of backup for personal files. DreamHost offers "unlimited" diskspace on their webservers for hosting files accessed by the web. However, my personal files aren't to be accessed by anyone but me and I just wanted an offsite backup.

I researched backup solutions, initially turning to my tried and true over the past 10+ years rsync scripts. However, I don't trust DreamHost to keep my data secure, so I *need* encryption. I narrowed down the choices to either a TrueCrypt image that I could mount in Linux and then split and rsync, or use duplicity.

I considered TrueCrypt for quite a while as the community following for it is rather impressive. However, syncing a single large image wasn't feasible. Splitting the 30-50GB image in to smaller pieces (guessing 250MB maybe) using the UNIX split command seemed to work with rsync, only transferring the major parts seemed to work on a 1GB test file I modified parts of. However, this just meant that now I needed to always keep 50GBs of space free so I could split the image which in turn was sync'd upstream.

I then resorted to duplicity. Initially, I didn't like idea of using tar (behind the scenes) which was then encrypted using GPG. I'm a long time users of rsync, where if I need one or two files I can instantly access them. However, I can't remember the last time I needed to that, so I bit the bullet and tried it. GPG is more secure as well (in my opinion) then TrueCrypt.

On Monday I gave it a shot, backing up my system's /etc directory and /home for the time being. The --dry-run in duplicity calculated approx 31GB of data to be copied... do some math using Wolfram Alpha considering my ~ 700Kb/s upload and we get 4 days 2 hours. Lovely, okay so my system will be hogging my Internet for a few days.

Some more research (via Google not firsthand) seems to suggest that duplicity won't pick-up where it left off either should it get interrupted, one of the features rsync does very well since it only deals with items on a file by file basis. So, I'll let it run.

After that I'll do incremental backups and then do a full backup next month. I also need to research compression algorithms in gpg's by adding "--gpg-options='--compress-algo=bzip2 --bzip2-compress-level=9'" to duplicity's options.

Oh yeah, and I need to make sure I can restore the backup.