As a computer programmer, It must have happened many times that you need a system or tool that sync two directories in real time and two way. We will go through one such tool mirror in this article. It is built to support two-way real-time sync between directory exists on the different machine. It is ideal for a two-machine (e.g. desktop+laptop) development workflow where you want to run a command line compile/build process on a powerful/dedicated desktop, but still edit files remotely on a laptop.
There are many tools available to sync directory like lsyncd, but what makes mirror unique is that it is two-way: it simultaneously syncs both laptop-to-desktop as well as desktop-to-laptop, in real time. Below is its comparison with similar existing tools available in the market.
Comparison to Existing Options
- unison is two-way, but not real-time
- rsync is not two-way, nor real-time
- lsyncd is real-time, but does not officially support two-way
- sshfs is too slow and doesn't support inotify
- doppleganger (an internal tool) is real-time, but not two-way
Non-Goals cum Limitations
- No Merge/long-duration disconnected support
- It will automatically re-connect and restart syncing when it detects the server is available again, but if files have changed on both sides while disconnected, then the last change will take priority.
- Maintain Unix permissions/owner/group
- Whatever Unix user runs the commands will be the owner/group/etc. of the files
- Not Support for huge files more than 100MB
- No Super-efficient diff/transmission logic like rsync
If a file changes, it retransmits the whole file instead of trying to diff only what changed.
Installation of mirror
Step 1: Install Java 8 or above. To check already installed java version, execute "java -version" command. If java is not installed, execute below command to install java 8.
1) On debian based operating system (Ubuntu, Debian etc.)
$ sudo apt-get update $ sudo apt-get install openjdk-8-jre
2) On RPM based operating system (RHEL, CentOS, Fedora etc.)
$ sudo yum update $ sudo yum install java-1.8.0-openjdk
Step 2 (Optional): Install watchman. The mirror will use the built-in Java WatchService API if you don't have watchman installed, and it will basically work, but the JDK-provided WatchService implementations are not that great: the Linux implementation is buggy.
Step 3: Download the latest mirror's executable to your home directory (or some other directory on your path, e.g. ~/bin)
$ wget http://repo.joist.ws/mirror-all.jar ~/ $ wget http://repo.joist.ws/mirror ~/
Step 4: Make the mirror file executable
$ chmod u+x mirror
Step 5: Copy both to your remote home directory (or some other directory on your path, e.g. ~/syncwork)
$ scp mirror-all.jar your-server.com:~/ $ scp mirror your-server.com:~/
Step 6: Start the server-side from the server's home directory
$ ./mirror server 2017-08-15 10:44:15 INFO Listening on 49172, version unspecified
Step 7: Now create a new directory in home directory with any name you like, I am creating a new directory named 'code'. Create one text file with some content in it, put it under code directory to test tool.
$ cd code $vi test.txt this is test.
Step 8: Start the client-side from the client's home directory
$ ./mirror client -h your-desktop.com -l ./code/ -r ./code/ 2017-08-15 10:47:04 INFO Connected, starting session, version unspecified 2017-08-15 10:47:04 ERROR Error creating watchman channel, skipping watchman java.lang.RuntimeException: Couldn't find watchman in possible paths [., /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin] with extensions [, .sh, .bat, .exe] at joist.util.Execute.resolve(Execute.java:182) at joist.util.Execute.getCommandPlusArgsArray(Execute.java:162) : at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2017-08-15 10:47:04 INFO Watchman not found, using WatchService instead 2017-08-15 10:47:04 INFO Note that WatchService is buggy on Linux, and uses polling on Mac. 2017-08-15 10:47:04 INFO While It will work with WatchService, especially to test, you should eventually install watchman. 2017-08-15 10:47:04 INFO Client has 2 paths 2017-08-15 10:47:04 INFO Server has 1 paths 2017-08-15 10:47:04 INFO Tree populated 2017-08-15 10:47:04 INFO Sending test.txt
As you can see from the last line in above log, mirror has sent the test.txt to the server. Great, We have successfully installed and tested mirror.
Note: Be careful using the tilde (e.g. ~/code), as your shell will resolve that, e.g. to /Users/you/code, and that resolved path on the client might be not valid on the server. This will sync the $HOME/code directory on your two machines.
Extra configuration in Mirror
By default, mirror will not sync any files in your listed in .gitignore file. However, you can also configure it with extra includes or excludes in addition to the .gitignore. Extra includes and excludes patterns can be passed when starting the client, and follow the .gitignore format, e.g.:
$ ./mirror client --exclude '.m2/` --include 'classpath' --include '.install' --exclude `build/`
Configure System Watch Limits
Note that if you have a lot of directories, you might have to increase the native file system limits, By default, max_user_watches limit is 8192, execute below command to increase it to 524288
$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
Using mirror for syncing More than Two Machines
Mirror also supports syncing more than two machines. E.g. you could have:
Server runs the mirror server process (you don't need to start multiple server processes)
Now suppose you want to sync one more machine other than server and client setup in the installation process, then on the new machine (Let's call it client 2), Perform step 8 of installation section to connect client 2 to the server.
Client 1 and Client 2 connects to the server and syncs it's ~/code to the server's ~/code
Now, All three machines(server, client 1, client 2) will be kept in sync. ~/code directory of all these three will be in sync. By the same approach, you can add any number of machines you want.
We have successfully installed, configured and tested mirror on Linux machine. Feel free to share any doubts or any new thing that world should know.
what about syncthing? seems to me same feature but more ergonomic and with a transparent connection of the machines
Hi Coo,
Yes, "syncthing" is very prominent. We have covered this topic 8 months ago. Here is the article. https://linoxide.com/ubuntu-how-to/setup-syncthing-synchronize-directories-ubuntu-16-04/