How To Backup Using Duplicity on Ubuntu 16.04

December 7, 2016 | By
| Reply More

Duplicity is a software application that provides encrypted, digitally signed, versioned tar-format backup volumes which can be uploaded to a remote or local file server. It is supported in all major Linux distribution like Ubuntu, Linux Mint, Debian, Fedora, and much more. Duplicity supports syncing files locally or remotely between filesystem or servers using, SSH/SCP, Rsync, FTP, WebDAV etc.

In this article, I'll explain on how to setup duplicity and use it to securely automate backups on your Ubuntu server.

Pre-requisites

We need to install all default packages for the Ubuntu from the repositories.

$apt-get update
$apt-get install ncftp python-paramiko python-pycryptopp lftp python-boto python-dev librsync-dev

Install Duplicity

We can install duplicity from the repository packages, by just running this command.

$ apt install duplicity

Let's confirm the duplicity version after install.

$ duplicity -V
duplicity 0.7.06

Create SSH and GPG Keys

Next, we will need to use SSH keys to securely authenticate with the remote system without having to provide a password. We also use GPG keys to encrypt the data before we transfer it to the backup location. These keys provide a secure interaction between the servers.

Let's generate an RSA 2048 bit encrypted SSH key from our root user to allow password-less logins to the backup machine.

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:gxdDFAjm6AUSi7LzczlDE1orygCzplmjQCqBFkxJ8Dc root@duplicity-01
The key's randomart image is:
+---[RSA 2048]----+
|O=o o. o+. |
|o*.= .. |
|Ooo E o |
|** = + . o |
|O.* + . S |
|BB + o . . |
|=.o = |
| o o |
| |
+----[SHA256]-----+

Now we copy this RSA key to our remote backup location.

$ssh-copy-id root@45.55.165.191
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '45.55.165.191 (45.55.165.191)' can't be established.
ECDSA key fingerprint is SHA256:XhFoQ3/mIsjGH7RfMwH6m0MHbj1B1kR4Sug5vfTQLdU.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@45.55.165.191's password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'root@45.55.165.191'"
and check to make sure that only the key(s) you wanted were added.

Hence, we can access the remote backup server without providing passwords from the host. We need to create a backup destination location to save our backups. I created a Duplicity folder on my remote backup server.

$mkdir -p remotebackup/Duplicity

Our next step is to create the GPG keys. GPG keys are used for extra security and encryption of the data transferred across the servers. Create it  by just running this command below:

$gpg --gen-key

This command will create the GPG keys and save it under  /root/.gnupg/ folder. This Key generation is interactive and we need to select and respond to various questions during the process.

$ gpg --gen-key
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection?
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Saheetha
Email address: saheetha1@gmail.com
Comment:
You selected this USER-ID:
"Saheetha <saheetha1@gmail.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

gpg: gpg-agent is not available in this session
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

Not enough random bytes available. Please do some other work to give
the OS a chance to collect more entropy! (Need 287 more bytes)

At this point, you are asked to generate entropy. Entropy is basically the unpredictability  in your system. Your VPS needs entropy to create a key that is actually random. If you receive this message as highlighted portion, you need to install haveged  package to create a key that is random and try the command again.

$apt-get install haveged

This package can be installed from your default repository. After installation, you need to make sure the following setting is updated as below in its configuration file /etc/default/haveged.

DAEMON_ARGS="-w 1024"

Finally, just make sure it's configured to start on boot:

$update-rc.d haveged defaults

After setting this, you can re-initiate the gpg --gen-key command once again and it will complete the process with the following.

$ gpg --gen-key
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection?
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N)
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Saheetha
Email address: yourname@gmail.com
Comment:
You selected this USER-ID:
"Saheetha <saheetha1@gmail.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

gpg: gpg-agent is not available in this session
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
+++++
...+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.................+++++
........+++++
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key F03B3360 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 2048R/F03B3360 2016-11-25
Key fingerprint = F262 785A B5E6 25E9 56E7 4484 3354 43D9 F03B 3360
uid Saheetha <yourname@gmail.com>
sub 2048R/C4D85223 2016-11-25

You can also get your gpg key information by running this command.

$ gpg --list-keys
/root/.gnupg/pubring.gpg
------------------------
pub 2048R/F03B3360 2016-11-25
uid Saheetha <yourname@gmail.com>
sub 2048R/C4D85223 2016-11-25

How to use Duplicity

Let's run an initial test for our duplicity system by creating a folder of dummy files to back up.  You can create a test folder and create some files to backup.

$ mkdir Test-DoC
$ cd Test-DoC
$ touch file{1..100}

Now, you can run this command to backup the server to our remote backup server.

$root@duplicity-01:~# duplicity /root/Test-DoC/ sftp://root@45.55.165.191//remotebackup/Duplicity
The authenticity of host '45.55.165.191' can't be established.
SSH-RSA key fingerprint is 20:9d:3b:fe:af:df:d1:40:5d:bc:f4:2c:6e:30:54:e0.
Are you sure you want to continue connecting (yes/no)? yes
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption:
Retype passphrase for decryption to confirm:
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1480059313.48 (Fri Nov 25 07:35:13 2016)
EndTime 1480059313.53 (Fri Nov 25 07:35:13 2016)
ElapsedTime 0.05 (0.05 seconds)
SourceFiles 101
SourceFileSize 4096 (4.00 KB)
NewFiles 101
NewFileSize 4096 (4.00 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 101
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 1029 (1.00 KB)
Errors 0
-------------------------------------------------

You can confirm the backup by login to our remote server.

root@duplicity-02:/remotebackup/Duplicity# ll
total 20
drwxr-xr-x 2 root root 4096 Nov 25 07:35 ./
drwxr-xr-x 3 root root 4096 Nov 25 07:11 ../
-rw-r--r-- 1 root root 438 Nov 25 07:35 duplicity-full.20161125T073505Z.manifest.gpg
-rw-r--r-- 1 root root 1029 Nov 25 07:35 duplicity-full.20161125T073505Z.vol1.difftar.gpg
-rw-r--r-- 1 root root 1596 Nov 25 07:35 duplicity-full-signatures.20161125T073505Z.sigtar.gpg

These files are expected to contain the backup information. Since it's just testing files we can remove the entire files.

Creating an Entire Server Backup

Let's create our first server backup. We are creating an entire root server backup excluding some of the folders like /proc, /sys, and /tmp. Here, we need to use the previously generated GPG pub key and Paraphrase to secure and encrypt our data. This is the general syntax for the backup creation.

duplicity --encrypt-key Pub-key_from_GPG --exclude files_to_exclude --include files_to_include path_to_back_up sftp://root@remotebackupHostname//remotebackup/duplicity

I've run this command to backup my server.

root@duplicity-01:~# PASSPHRASE="docker" duplicity --encrypt-key F03B3360 --exclude /proc --exclude /sys --exclude /tmp  / sftp://root@45.55.165.191//remotebackup/Duplicity/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1480143194.50 (Sat Nov 26 06:53:14 2016)
EndTime 1480143304.84 (Sat Nov 26 06:55:04 2016)
ElapsedTime 110.34 (1 minute 50.34 seconds)
SourceFiles 69101
SourceFileSize 885736795 (845 MB)
NewFiles 69101
NewFileSize 885736795 (845 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 69101
RawDeltaSize 847552198 (808 MB)
TotalDestinationSizeChange 345904578 (330 MB)
Errors 0
-------------------------------------------------

It will take some time to complete this backup process. Because this is the first time we've run the backup, duplicity will create a full backup. Duplicity divides the chunks of data into volumes to simplify the file transfers.

root@duplicity-02:/remotebackup/Duplicity# ll
total 354864
drwxr-xr-x 2 root root 4096 Nov 26 06:55 ./
drwxr-xr-x 3 root root 4096 Nov 25 07:11 ../
-rw-r--r-- 1 root root 303666 Nov 26 06:55 duplicity-full.20161126T065314Z.manifest.gpg
-rw-r--r-- 1 root root 26241374 Nov 26 06:54 duplicity-full.20161126T065314Z.vol10.difftar.gpg
-rw-r--r-- 1 root root 26192752 Nov 26 06:54 duplicity-full.20161126T065314Z.vol11.difftar.gpg
-rw-r--r-- 1 root root 26206134 Nov 26 06:54 duplicity-full.20161126T065314Z.vol12.difftar.gpg
-rw-r--r-- 1 root root 26196266 Nov 26 06:54 duplicity-full.20161126T065314Z.vol13.difftar.gpg
-rw-r--r-- 1 root root 4999361 Nov 26 06:55 duplicity-full.20161126T065314Z.vol14.difftar.gpg
-rw-r--r-- 1 root root 26256306 Nov 26 06:53 duplicity-full.20161126T065314Z.vol1.difftar.gpg
-rw-r--r-- 1 root root 26216804 Nov 26 06:53 duplicity-full.20161126T065314Z.vol2.difftar.gpg
-rw-r--r-- 1 root root 26198345 Nov 26 06:53 duplicity-full.20161126T065314Z.vol3.difftar.gpg
-rw-r--r-- 1 root root 26197666 Nov 26 06:53 duplicity-full.20161126T065314Z.vol4.difftar.gpg
-rw-r--r-- 1 root root 26237799 Nov 26 06:53 duplicity-full.20161126T065314Z.vol5.difftar.gpg
-rw-r--r-- 1 root root 26218126 Nov 26 06:53 duplicity-full.20161126T065314Z.vol6.difftar.gpg
-rw-r--r-- 1 root root 26252966 Nov 26 06:53 duplicity-full.20161126T065314Z.vol7.difftar.gpg
-rw-r--r-- 1 root root 26234136 Nov 26 06:54 duplicity-full.20161126T065314Z.vol8.difftar.gpg
-rw-r--r-- 1 root root 26256543 Nov 26 06:54 duplicity-full.20161126T065314Z.vol9.difftar.gpg
-rw-r--r-- 1 root root 17136137 Nov 26 06:55 duplicity-full-signatures.20161126T065314Z.sigtar.gpg

On a fresh droplet, my configuration created 15 volumes which were transferred to the remote system. Since we have a full backup for my server on our backup server, next backup will  be an incremental backup. These are faster and required less data transfer time. You can even force another full backup by just running this command.

$PASSPHRASE="docker" duplicity full --encrypt-key F03B3360 --exclude /proc --exclude /sys --exclude /tmp  / sftp://root@45.55.165.191//remotebackup/Duplicity/

Scheduling our Backups

Now we can learn on how to schedule our backups daily or weekly by setting the backup script to run automatically. First of all, let us create a passphrase file under our /root folder to pass the passphrase automatically without an interactive prompt for paraphrase.

$cat /root/.passphrase
PASSPHRASE="docker"

Secure the file permissions and restrict to 600.

Let's see how to create the backup daily. We need to create our backup script inside the /etc/cron.daily folder. All the cron jobs set inside this folder will run daily.

$:/etc/cron.daily# cat duplicity.inc
#!/bin/sh

test -x $(which duplicity) || exit 0
. /root/.passphrase

export PASSPHRASE
$(which duplicity) --encrypt-key F03B3360 --exclude /proc --exclude /sys --exclude /tmp --exclude /var / sftp://root@45.55.165.191//remotebackup/Duplicity/

Make sure to give executable permission for the script.

$:/etc/cron.daily# chmod +x duplicity.inc

We can run this script to make sure everything works properly.

root@duplicity-01:/etc/cron.daily# ./duplicity.inc
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Nov 26 06:53:14 2016
--------------[ Backup Statistics ]--------------
StartTime 1480564726.91 (Thu Dec 1 03:58:46 2016)
EndTime 1480564748.83 (Thu Dec 1 03:59:08 2016)
ElapsedTime 21.91 (21.91 seconds)
SourceFiles 69108
SourceFileSize 899520535 (858 MB)
NewFiles 32
NewFileSize 20761841 (19.8 MB)
DeletedFiles 13
ChangedFiles 9
ChangedFileSize 2779 (2.71 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 54
RawDeltaSize 20737072 (19.8 MB)
TotalDestinationSizeChange 17502790 (16.7 MB)
Errors 0
-------------------------------------------------

Similarly, we can schedule a backup to force a full backup weekly and maintain. Since incremental backups will get increasingly unwieldy as changes stack up. We will configure weekly full backups to refresh the base.

Let's do this by creating a duplicity full backup script under the /etc/cron.weekly folder. You can create a duplicity.full script inside this directory.

$:/etc/cron.weekly# chmod +x duplicity.full
$:/etc/cron.weekly# cat duplicity.full
#!/bin/sh

test -x $(which duplicity) || exit 0
. /root/.passphrase

export PASSPHRASE
$(which duplicity) full --encrypt-key F03B3360 --exclude /proc --exclude /sys --exclude /tmp --exclude /var / sftp://root@45.55.165.191//remotebackup/Duplicity/

We can also append this command to our duplicity.full script to clean out the old backup files. Add this command to retain only three backups and their corresponding incremental backups.

$(which duplicity) remove-all-but-n-full 3 --force sftp://root@45.55.165.191//remotebackup/Duplicity/

Hence, our weekly script will create a full backup for our server and remove all unwanted old backup files from the server. We can run our script manually to test its working.

$:/etc/cron.weekly# ./duplicity.full
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Nov 26 06:53:14 2016
--------------[ Backup Statistics ]--------------
StartTime 1480570674.29 (Thu Dec 1 05:37:54 2016)
EndTime 1480570791.54 (Thu Dec 1 05:39:51 2016)
ElapsedTime 117.26 (1 minute 57.26 seconds)
SourceFiles 69109
SourceFileSize 906577560 (865 MB)
NewFiles 69109
NewFileSize 906577560 (865 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 69109
RawDeltaSize 868388867 (828 MB)
TotalDestinationSizeChange 363488138 (347 MB)
Errors 0
-------------------------------------------------

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Dec 1 05:37:53 2016
No old backup sets found, nothing deleted.

Conclusion

Congratulations! We've our  fully operational automated backup solution ready to use. One of the main advantages of Duplicity over other backup solutions is that it encrypts the data using the GnuPG key. It also provides an automated backup solution which is great and can be done using simple Cron jobs. I hope this article is useful for you! Please post your valuable comments and suggestions on this.

Filed Under : LINUX HOWTO

Tagged With : ,

Free Linux Ebook to Download

Leave a Reply

All comments are subject to moderation.