How to Setup GitLab Server Backup

November 24, 2016 | By
| Reply More

GitLab is an absolute solution for storing all your Git repositories. Taking backup is an important activity even when a few people ignore it because taking backups repeatedly are too tiresome and adds to their frustration as a developer. So, automating the whole process of taking the backup of your GitLab repositories is a good idea. I'll explain here how to achieve that.

In my previous article, I've explained on how to install a Secure GitLab on an Ubuntu server. In this article, I'll explain on how to backup its data. One basic approach that everyone should follow is to take backup daily, weekly and monthly because if one fails you have another.  Never keep your backup on the GitLab server itself because if server crashes then your backups are lost too. Let's see how to backup your GitLab server and store on a remote server.

Creating Backup

GitLab has a built-in task to backup its data and repositories. But we need to make sure that we run this backup script as git user to maintain a consistent and proper file permissions.

root@linoxide:/home/git/gitlab# sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
done
Dumping repositories ...
done
Dumping uploads ...
done
Creating backup archive: 1479023906_gitlab_backup.tar ... done
Deleting tmp directories ... done
Deleting old backups ... skipping

The backup is created in the "/home/git/gitlab/tmp/backups"  folder under the gitlab installation.

root@linoxide:/home/git/gitlab/tmp/backups# ll
total 68
drwxr-xr-x 2 git git 4096 Nov 13 07:58 ./
drwxr-xr-x 7 git git 4096 Nov 13 07:58 ../
-rw-r--r-- 1 git git 61440 Nov 13 07:58 1479023906_gitlab_backup.tar

Backup the Configuration files

GitLab backup utility backups only the data and repositories. But as a safety measure, it is always advised to keep the backup for the configuration files  which can be used while restoring a GitLab application backup. I've preferred to backup the entire "GitLab" config folder. I created a backup.sh script to backup the GitLab configuration folder and save it under our GitLab backup folder "/home/git/gitlab/tmp/backups/config-backups/".  Please see my backup script below:

#!/bin/sh
umask 0077;
cd /home/git/gitlab/
tar -czf /home/git/gitlab/tmp/backups/config-backups/config.$(date +%s).tgz config

We can add this script to run in Cron to backup these configuration files automatically as required. I've scheduled it to run every Monday 2AM. I'll explain about this Cron in the later section.

Enabling SSH key Authentication between the servers

We need to create an SSH key for root on GitLab source and copy the public key over to the GitLab backup server to enhance the SSH connection between the servers during the account sync.

GitLab SRC Server’s IP = 104.131.169.217
GitLab DEST  Server’s IP = 104.236.195.186

root@linoxide:/home/git/gitlab/tmp/backups# ssh-keygen
root@linoxide:# ssh-copy-id -i /root/.ssh/id_rsa.pub root@104.236.195.186

Once, it's done, you can verify the authorization keys on the Backup server.

root@-backupnode:~/.ssh# cat authorized_keys

Syncing Backup files

Here we are using live syncing daemon named as lsyncd. It is used to sync or replicate files & directories locally and remotely after a specific interval of time. It uses Rsync & SSH in the backend. It monitors the directory on the source server, if any changes or modification occur on the source end, then lsyncd will replicate the same on its Backup server after a specific interval of time.

We just need to install this on the Source Server. You can install this package with just one command.

root@linoxide:/homes# apt-get install lsyncd

Next, we need to set up the configuration file of lsyncd to make it working as required. First of all, we need to copy the lrsync.lua file from the "/usr/share" folder and copy it over to /etc folder.

I created a "lsyncd" folder inside /etc/ folder and copy this file over there.

root@linoxide-:/etc# mkdir /etc/lsyncd
root@linoxide-:/etc# cp -rp /usr/share/doc/lsyncd/examples/lrsync.lua /etc/lsyncd/lsyncd.conf.lua

Add the following lines to the configuration file as in the screenshot.

lsyncd-configuration

We will change "default.rsync" to "default.rsyncssh" to enable rsync over ssh, and we should replace the "target" variable with the "host" and "targetdir" variables. In addition, we need to create a log directory and log/status files for lsyncd to use.

root@linoxide-:/etc# mkdir -p /var/log/lsyncd/
root@linoxide-:/var/log/lsyncd# touch lsyncd.log lsyncd-status.log

Finally, we need to restart and enable the lsyncd service on the source server.

root@linoxide-:/etc/lsyncd#systemctl restart lsyncd
root@linoxide-:/etc/lsyncd# systemctl status lsyncd

Configuring Cron jobs

Our next task is to configure cronjobs on the source server to generate Git repository backup files. We can even create one more cronjob to clear older backups from the Backup folder. Let's automate all of this by using Cron jobs.

I've created three Cronjobs as follows :

# m h dom mon dow command
0 1 * * 1 cd /home/git/gitlab && sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
0 3 * * 1 find /home/git/gitlab/tmp/backups/*.tar -mtime +30 -exec rm {} \;
0 2 * * 1 /root/backup.sh

  • To create the GitLab data and application backup at 1AM every week Monday
  • To  remove all backups which are older than 30 days from the backups folder.
  • To backup the configuration folder every Monday at 2AM

I've added these to my Root Cron using the command "crontab -e -u root".

Please Note : Any file changes in the GitLab backup folder will be automatically updated to our Remote Backup server.

Verifying the Backups

Finally, I've verified our Backups on the backup server by login to my remote Backup server.

root@backupnode:/home/gitlab/backup# pwd
/home/gitlab/backup

root@backupnode:/home/gitlab/backup# ls -R
.:
1479107412_gitlab_backup.tar 1479108008_gitlab_backup.tar 1479108610_gitlab_backup.tar config-backups
1479107710_gitlab_backup.tar 1479108309_gitlab_backup.tar 1479108912_gitlab_backup.tar

./config-backups:
config.1479189401.tgz config.1479189661.tgz config.1479189781.tgz config.1479189901.tgz
config.1479189601.tgz config.1479189721.tgz config.1479189841.tgz

The backup files on both the servers remain exactly same.

Conclusion

This should be the basics of how to keep your GitLab server backed up. This is the most simple way to backup your GitLab server. Every day new data keeps adding to your GitLab server. But you don't realize that your server can crash due to a lot of reasons and sometimes by the time you realize all your data is lost, all your repositories are now residing somewhere you obviously don't know. So, taking backup of your applications are as important as adding new features to them. It is a part of maintaining your application which is generally ignored by most of the people. So keep your data safe & take backup periodically.

Filed Under : LINUX HOWTO

Tagged With : , ,

Free Linux Ebook to Download

Leave a Reply

Commenting Policy:
Promotion of your products ? Comment gets deleted.
All comments are subject to moderation.