How to Setup lsyncd for Synchronize Directories to Remote Computers

Lsyncd is a synchronization daemon that enables you to synchronizes your designated directory with to any other directory on the network or locally. ie Its helps mirror local directories with remote targets

Software like csync, rsync, csync2, mirror will all do the job but in this article, we will show installation and configuration of lsyncd. To save network and disk bandwidth, it only mirror changes to your directory using Rsync+ssh.

Installing lsyncd

It is present in Ubuntu repositories so we can install it by simple apt command

sudo apt install lsyncd

For CentOS you first need to install EPEL repository because lsyncd is there.

yum install epel-release

And next we can install Live Syncing Daemon which is actual long name of the program

yum install lsyncd

It will not configure anything to sync, you need to do this by yourself and we will show how. Sample configuration files live in /usr/share/doc/lsyncd/examples/ directory in ubuntu and /usr/share/doc/lsyncd-2.1.5/examples/ in centos so lets check what is there

[email protected]:~$ ls /usr/share/doc/lsyncd*/examples/
lbash.lua lgforce.lua lpostcmd.lua lrsyncssh.lua
lecho.lua limagemagic.lua lrsync.lua

Those are sample config files for lsyncd. They are all written in lua same as lsyncd and they are interpreted when sync job is ran. Lets first create directories with which we work.

Setting up local syncing with lsyncd

First we create the directory which we will sync from, that is, the source directory:

sudo mkdir -p /linoxide/sync/

And then the directory which will be a backup of this one, that is, the target directory

sudo mkdir /backup

Next lets populate the source directory with some files

sudo touch /linoxide/sync/testfile{1..25}

We would want to have some logging, so lets make file for log and status

sudo mkdir /var/log/lsyncd
sudo touch /var/log/lsyncd/lsyncd.{log,status}

Next we need to make a dir in /etc directory for the configuration file

sudo mkdir /etc/lsyncd

There we create new configuration file from scratch

sudo nano /etc/lsyncd/lsyncd.conf.lua

In this config file you can paste the below code.

settings {
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd.status"
}

sync {
        default.rsync,
        source = "/linoxide/sync/",
        target = "/backup",
}

Then we can go to target directory and check of something is there

cd /backup/

ls

It should give you no output. When we restart the lsyncd service, it should immediately sync with the directory.

sudo systemctl restart lsyncd

[email protected]:/backup$ ls
testfile1 testfile14 testfile19 testfile23 testfile5
testfile10 testfile15 testfile2 testfile24 testfile6
testfile11 testfile16 testfile20 testfile25 testfile7
testfile12 testfile17 testfile21 testfile3 testfile8
testfile13 testfile18 testfile22 testfile4 testfile9

The sync is working. Lets then add some more files to our source dir, to se if it syncs  on the fly.

[email protected]:/backup$ sudo touch /linoxide/sync/new{1..9}

It doesn't sync every second for performance reasons, so we need to wait some time and then type ls in our /backup folder to see if new files are added

[email protected]:/backup$ ls
new1 new6 testfile10 testfile15 testfile2 testfile24 testfile6
new2 new7 testfile11 testfile16 testfile20 testfile25 testfile7
new3 new8 testfile12 testfile17 testfile21 testfile3 testfile8
new4 new9 testfile13 testfile18 testfile22 testfile4 testfile9
new5 testfile1 testfile14 testfile19 testfile23 testfile5

The files are there.

Syncing to remote location

If you want your target or source directory to be on remote machine, you first need to setup SSH keys for passwordless ssh login. If you didn't already create your ssh key, do so now by following command. Make sure you do it as root as lsyncd will also try to log in as root. Using normal user will not work so we use su

su

ssh-keygen -t rsa

You press enter to all prompts, and when it is finished, you copy the key to target machine by the following command

ssh-copy-id [email protected]

Off course, change the IP address for your target machine. It will prompt you for the password on that machine for the last time, and from now on you will log in there without password. So  lets login to our remote server:

ssh [email protected]

There we will create a directory that will be our remote target:

mkdir /remotebackup

On our source machine that is running lsyncd we need to edit config file to make it look like this

settings {

insist = true,
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status"
}

sync {
default.rsyncssh,
delete=true,
source = "/linoxide/sync/",
host = "159.203.161.112",
targetdir = "/remotebackup"
exclude={'.tmp', 'wp-content/cache'},
 rsync = {
     update = true,
     archive = true,
     links = false,
     perms = true,
     owner = true,
     compress = false,
     whole_file = false
   }
}

insist = true // lsyncd service will keep running even one or more targets are not reachable at startup

delete=true // If u delete file or directory on source it will be deleted on target as well

exclude={'.tmp', 'wp-content/cache'} // List of strings will be excluded from syncing

// linux rsync options can be called using below settings

rsync = {
update = true,
archive = true,
links = false,
perms = true,
owner = true,
compress = false,
whole_file = false
}

Next cd into /remotebackup on the server and check if it is empty. Then restart the lsyncd on your source machine with systemctl restart lsyncd command and look if it syncs on the server.

ls

syncing works

Syncing the files over multiple targets

Let's see how we can sync the files from a source server over multiple targets using lsyncd. I've taken three Ubuntu servers for this purpose. One as my source server and the other two as our target servers. To allow passwordless login to the servers during this rsync process, we need to create SSH keys on the source server and copy it over the destination servers.

Please see the details below:

Source Server : 45.79.81.125
Target Server 1 : 45.79.82.74
Target Server 2 : 45.79.106.109

First of all you need to login to your source server and install lsyncd using the command below:

#apt install lsyncd

Now I've created my source directory at /linoxide/source-target/ on the source server using this command below:

# mkdir -p /linoxide/source-target/

Next let's populate the source directory with some files:

#touch /linoxide/source-target/testfile{1..25}

And then move on to the remote servers and create our target directory which will be a backup of this one.

#mkdir /backup-dest1 *// On Target Server 1 //*
#mkdir /backup-dest2  *// On Target Server 2//*

To confirm the lsyncd status, make sure to create the status file and log file at our convenient location. I've created them as below:

#mkdir /var/log/lsyncd
#touch /var/log/lsyncd/lsyncd.{log,status}

Finally, I need to modify the lsyncd configuration to sync this source directory to my remote destination locations at /backup-dest1 and /backup-dest2 on the target server 1 and target server 2 respectively. It will not configure anything to sync by default, you need to do the basic configuration by yourself. You can view all sample configuration files live in /usr/share/doc/lsyncd/examples/ directory. I've appended two of the configuration files from that location namely lrsyncssh.lua and lrsync.lua which suit my purpose and modified accordingly. Please see my lsyncd configuration file below:

# cat /etc/lsyncd/lsyncd.conf.lua
 settings {
 logfile = "/var/log/lsyncd/lsyncd.log",
 statusFile = "/var/log/lsyncd/lsyncd-status.log",
 statusInterval = 1,
 }
 sync{default.rsyncssh, source="/linoxide/source-target/", host="45.79.82.74", targetdir="/backup-dest1"}
 sync{default.rsyncssh, source="/linoxide/source-target/", host="45.79.106.109", targetdir="/backup-dest2"}

You can add any number of multiple targets to this configuration with just adding/modifying the sync command entries as shown in the configuration with the new host info.

When we restart the lsyncd service, it should immediately sync the directory to the remote locations.

# tail -f /var/log/lsyncd/lsyncd.log
 Thu Nov 2 11:34:48 2017 Normal: recursive startup rsync: /linoxide/source-target/ -> 45.79.82.74:/backup-dest1/
 Thu Nov 2 11:34:48 2017 Normal: recursive startup rsync: /linoxide/source-target/ -> 45.79.106.109:/backup-dest2/
 Thu Nov 2 11:34:48 2017 Normal: Startup of "/linoxide/source-target/" finished: 0
 Thu Nov 2 11:34:48 2017 Normal: Startup of "/linoxide/source-target/" finished: 0

Few more commands

Check if lsyncd starts without errors

sudo /usr/bin/lsyncd -nodaemon -log all /etc/lsyncd/lsyncd.conf.lua

or

lsyncd /etc/lsyncd/lsyncd.conf.lua

Start synchronization

sudo /usr/bin/lsyncd /etc/lsyncd/lsyncd.conf.lua

Add to root's crontab to automate startup

@reboot keep-one-running /usr/bin/lsyncd /etc/lsyncd/lsyncd.conf.lua

Conclusion

We have set up local and remote syncing with live syncing daemon, but that is just scratch on the surface of what lsyncd can do. If you look at /usr/share/doc/lsyncd/examples/ directory for example configurations, you can see that this program can be used for example to continuously fix user permissions on all files in some directory and more. This is all for this article, thank you for reading and have a good day.

Mihajlo Milenovic 4:26 am

About Mihajlo Milenovic

Miki is a long time GNU/Linux user, Free Software advocate and a freelance system administrator from Serbia. Got introduced to GNU/Linux in year 2003 on old AMD Duron computer, and since than always eager to learn new stuff about this system. From 2016 writes for Linoxide to share his experiences with wider audience

Author Archive Page

Have anything to say?

Your email address will not be published. Required fields are marked *

All comments are subject to moderation.

2 Comments

  1. Thank you for your manual, very usefull for me.

    How to edit /etc/lsyncd/lsyncd.conf.lua if I want to backup my files on three or more servers simultaneously? Moreover, every remote server has its own specific dir path where to backup my data

    Your example described only a single remote backup server.

    Thank you!