How to Install "mirror" for Real Time and Two Way Sync

As a computer programmer, It must have happened many times that you need a system or tool that sync two directories in real time and two way. We will go through one such tool mirror in this article. It is built to support two-way real-time sync between directory exists on the different machine. It is ideal for a two-machine (e.g. desktop+laptop) development workflow where you want to run a command line compile/build process on a powerful/dedicated desktop, but still edit files remotely on a laptop.

There are many tools available to sync directory like lsyncd, but what makes mirror unique is that it is two-way: it simultaneously syncs both laptop-to-desktop as well as desktop-to-laptop, in real time. Below is its comparison with similar existing tools available in the market.

Comparison to Existing Options

• unison is two-way, but not real-time
• rsync is not two-way, nor real-time
• lsyncd is real-time, but does not officially support two-way
• sshfs is too slow and doesn't support inotify
• doppleganger (an internal tool) is real-time, but not two-way

Non-Goals cum Limitations

• No Merge/long-duration disconnected support

It will automatically re-connect and restart syncing when it detects the server is available again, but if files have changed on both sides while disconnected, then the last change will take priority.

• Maintain Unix permissions/owner/group

Whatever Unix user runs the commands will be the owner/group/etc. of the files

• Not Support for huge files more than 100MB
• No Super-efficient diff/transmission logic like rsync

If a file changes, it retransmits the whole file instead of trying to diff only what changed.

Installation of mirror

Step 1: Install Java 8 or above. To check already installed java version, execute "java -version" command. If java is not installed, execute below command to install java 8.

1) On debian based operating system (Ubuntu, Debian etc.)

$ sudo apt-get update
$ sudo apt-get install openjdk-8-jre

2) On RPM based operating system (RHEL, CentOS, Fedora etc.)

$ sudo yum update
$ sudo yum install java-1.8.0-openjdk

Step 2 (Optional): Install watchman. The mirror will use the built-in Java WatchService API if you don't have watchman installed, and it will basically work, but the JDK-provided WatchService implementations are not that great: the Linux implementation is buggy.

Step 3: Download the latest mirror's executable to your home directory (or some other directory on your path, e.g. ~/bin)

$ wget http://repo.joist.ws/mirror-all.jar ~/
$ wget http://repo.joist.ws/mirror ~/

Step 4: Make the mirror file executable

 $ chmod u+x mirror

Step 5: Copy both to your remote home directory (or some other directory on your path, e.g. ~/syncwork)

$ scp mirror-all.jar your-server.com:~/
$ scp mirror your-server.com:~/

Step 6: Start the server-side from the server's home directory

$ ./mirror server
2017-08-15 10:44:15 INFO  Listening on 49172, version unspecified

Step 7: Now create a new directory in home directory with any name you like, I am creating a new directory named 'code'. Create one text file with some content in it, put it under code directory to test tool.

$ cd code
$vi test.txt
this is test.

Step 8: Start the client-side from the client's home directory

$ ./mirror client -h your-desktop.com -l ./code/ -r ./code/
2017-08-15 10:47:04 INFO  Connected, starting session, version unspecified
2017-08-15 10:47:04 ERROR Error creating watchman channel, skipping watchman
java.lang.RuntimeException: Couldn't find watchman in possible paths [., /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin] with extensions [, .sh, .bat, .exe]
	at joist.util.Execute.resolve(Execute.java:182)
	at joist.util.Execute.getCommandPlusArgsArray(Execute.java:162)
	:
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

2017-08-15 10:47:04 INFO  Watchman not found, using WatchService instead
2017-08-15 10:47:04 INFO    Note that WatchService is buggy on Linux, and uses polling on Mac.
2017-08-15 10:47:04 INFO    While It will work with WatchService, especially to test, you should eventually install watchman.
2017-08-15 10:47:04 INFO  Client has 2 paths
2017-08-15 10:47:04 INFO  Server has 1 paths
2017-08-15 10:47:04 INFO  Tree populated
2017-08-15 10:47:04 INFO  Sending test.txt

As you can see from the last line in above log, mirror has sent the test.txt to the server. Great, We have successfully installed and tested mirror.

Note: Be careful using the tilde (e.g. ~/code), as your shell will resolve that, e.g. to /Users/you/code, and that resolved path on the client might be not valid on the server. This will sync the $HOME/code directory on your two machines.

Extra configuration in Mirror

By default, mirror will not sync any files in your listed in .gitignore file. However, you can also configure it with extra includes or excludes in addition to the .gitignore. Extra includes and excludes patterns can be passed when starting the client, and follow the .gitignore format, e.g.:

$ ./mirror client --exclude '.m2/` --include 'classpath' --include '.install' --exclude `build/`

Configure System Watch Limits

Note that if you have a lot of directories, you might have to increase the native file system limits, By default, max_user_watches limit is 8192, execute below command to increase it to 524288

$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Using mirror for syncing More than Two Machines

Mirror also supports syncing more than two machines. E.g. you could have:

Server runs the mirror server process (you don't need to start multiple server processes)

Now suppose you want to sync one more machine other than server and client setup in the installation process, then on the new machine (Let's call it client 2), Perform step 8 of installation section to connect client 2 to the server.
Client 1 and Client 2 connects to the server and syncs it's ~/code to the server's ~/code

Now, All three machines(server, client 1, client 2) will be kept in sync. ~/code directory of all these three will be in sync. By the same approach, you can add any number of machines you want.

We have successfully installed, configured and tested mirror on Linux machine. Feel free to share any doubts or any new thing that world should know.