On Linux operating system, the 'rsync' stands for Remote Sync. It's an utility used to synchronize (copy) files and directories from a source (SRC) to a destination (DEST).
Files and directories could be synced on a local host or to/from a remote host. On the SRC, files and directories will be synchronized. Whereas, on the DEST, the synced files and directories will be taken place.
In addition, rsync command can recursively copy files and directories, copy symbolic links, preserve (permissions, group, modification time and ownership) file identity.
Rsync can use two different ways for syncing/copying files or directories:
- Use remote shell: ssh or rsh
- Use rsync daemon directly through TCP
Even more, rsync uses delta-transfer algorithm which only copy/sync the different data between SRC and DEST.
This tutorial will go through many ways to use rsync command through hands-on practices and examples. We will also explain the detailed of the most common options of rsync command.
Rsync command syntax
Before deep-diving into the ways of using rsync command, let's take a look at its syntax:
rsync [options] <SRC_PATH> <DEST_PATH>
1. Copy/Sync file on the local machine
Rsync command can be used to copy/sync file on your local machine.
Assuming that you want to copy a file from home directory of user 'linoxide' named '/home/linoxide' to folder '/opt/test'.
$ sudo rsync -zvh /home/linoxide/computer-networking.pdf /opt/test/ computer-networking.pdf sent 1.01K bytes received 35 bytes 2.10K bytes/sec total size is 3.03K speedup is 2.89
-v: verbose the output
2. Copy/Sync directory on the local machine
To transfer all the files from SRC local directory to DEST directory, run the rsync command with option
Assuming that you want to copy home directory of user 'linoxide' named '/home/linoxide' to folder '/opt/backup'.
$ sudo rsync -zavh /home/linoxide/ /opt/backup/ sending incremental file list ./ .bash_history .bash_logout .bashrc .lesshst .profile .sudo_as_admin_successful .viminfo computer-networking.pdf .ssh/ .ssh/known_hosts sent 4.85K bytes received 198 bytes 10.10K bytes/sec total size is 9.59K speedup is 1.90
-a: archive data during the copy/sync process
You can verify whether the rsync command was successful or not? Run the below command:
$ ls /opt/backup/ -la total 40 drwxr-xr-x 3 linoxide linoxide 4096 Thg 6 15 16:02 . drwxr-xr-x 4 root root 4096 Thg 6 15 16:25 .. -rw------- 1 linoxide linoxide 817 Thg 6 15 16:01 .bash_history -rw-r--r-- 1 linoxide linoxide 220 Thg 6 11 15:58 .bash_logout -rw-r--r-- 1 linoxide linoxide 3771 Thg 6 11 15:58 .bashrc -rw-rw-r-- 1 linoxide linoxide 3028 Thg 2 25 2017 computer-networking.pdf -rw------- 1 linoxide linoxide 40 Thg 6 15 15:51 .lesshst -rw-r--r-- 1 linoxide linoxide 807 Thg 6 11 15:58 .profile drwx------ 2 linoxide linoxide 4096 Thg 6 13 17:58 .ssh -rw-r--r-- 1 linoxide linoxide 0 Thg 6 15 16:02 .sudo_as_admin_successful -rw------- 1 linoxide linoxide 680 Thg 6 14 16:50 .viminfo
3. Rsync over ssh with specific Port
You can use ssh (secure shell) to copy/sync data, your data will be transferred in a secure connection with encryption. Nobody can read your data while it is being transferred on the Internet. When you use rsync command, you need to provide the user/root password to accomplish the particular task. Using the SSH option will send your logins in an encrypted manner so that your password will be safe.
Copy a file from a remote to local with a specific ssh port. In order to specify a protocol with rsync command, you need to use
-e option with protocol name you want to use.
For example, the following rsync command will copy file 'deploy.yaml' via a remote shell from a remote machine to home directory of user 'linoxide'.
$ rsync -avz -e "ssh -p 22" firstname.lastname@example.org:/home/cas/deploy.yaml /home/linoxide/ email@example.com's password: receiving incremental file list sent 20 bytes received 60 bytes 32.00 bytes/sec total size is 239 speedup is 2.99
4. Copy/Sync files and directories from local machine to remote machine
Supposing that you want to copy/sync the folder 'test-data' in home directory of user 'linoxide' in local host to home directory of a remote machine 'firstname.lastname@example.org', run command:
$ rsync -azrvh /home/linoxide/computer-networking.pdf email@example.com: firstname.lastname@example.org's password: sending incremental file list computer-networking.pdf sent 1.04K bytes received 35 bytes 430.80 bytes/sec total size is 3.03K speedup is 2.81
-r: recursive copy
5. Copy/Sync files and directories from remote machine to local machine
The rsync command helps you transfer a remote directory 'document' to be copied in your local host.
Assuming that you want to copy/sync files/directories from a remote host 'email@example.com' to home directory of user 'linoxide'.
$ rsync -zavhr firstname.lastname@example.org:document /home/linoxide email@example.com's password: receiving incremental file list document/ document/kubernetes.txt sent 47 bytes received 139 bytes 74.40 bytes/sec total size is 0 speedup is 0.00
6. Rsync -include and -exclude options
These two options allow us to include and exclude files by specifying parameters with these option helps us to specify those files or directories which you want to include in your sync and exclude files and folders with you don’t want to be transferred.
Here in this example, rsync command will include those files and directory only which starts with 'k' and exclude all other files and directory.
$ rsync -avhz -e "ssh -p 22" --include 'k*' --exclude '*' firstname.lastname@example.org: /home/linoxide email@example.com's password: receiving incremental file list ./ k8s.yaml kubernetes.yaml kuber/ sent 88 bytes received 280 bytes 147.20 bytes/sec total size is 73 speedup is 0.20
7. Set max size of files
You can specify the max file size to be transferred or sync. You can do it with '--max-size' option.
In the following example, max file size is 200k, so this command will transfer only those files which are equal or smaller than 200k.
$ rsync -zavhr --max-size='200k' firstname.lastname@example.org: /home/linoxide email@example.com's password: receiving incremental file list .bash_history .bash_logout .bashrc .profile .sudo_as_admin_successful .viminfo computer-networking.pdf sent 2.58K bytes received 51.60K bytes 21.67K bytes/sec total size is 113.24K speedup is 2.09
8. Delete source files after successful transfer
Now, suppose you have a web server and a data backup server, you created a daily backup and synced it with your backup server, now you don’t want to keep that local copy of backup in your web server. So, will you wait for the transfer to complete and then delete that local backup file manually? Of Course NO. This automatic deletion can be done using '--remove-source-files' option.
$ ls /home/directory computer-networking.pdf deploy.yaml document k8s.yaml kuber kubernetes.yaml ------ $ rsync -azvh --remove-source-files /home/linoxide/deploy.yaml firstname.lastname@example.org: email@example.com's password: sending incremental file list sent 70 bytes received 20 bytes 36.00 bytes/sec total size is 239 speedup is 2.66
Run the 'ls' command again, you can see the file 'deploy.yaml' was deleted on SRC.
$ ls computer-networking.pdf document k8s.yaml kuber kubernetes.yaml
9. Find the difference in files and directories between SRC and DEST
In order to find any difference in the files of directories between SRC and DEST, you can run rsync command with '-i' option.
$ rsync -avzi /home/linoxide/cloud firstname.lastname@example.org: email@example.com's password: sending incremental file list cd+++++++++ cloud/ <f+++++++++ cloud/computer-networking.pdf <f+++++++++ cloud/k8s.yaml <f+++++++++ cloud/kubernetes.yaml cd+++++++++ cloud/document/ <f+++++++++ cloud/document/kubernetes.txt cd+++++++++ cloud/kuber/ <f+++++++++ cloud/kuber/test.txt sent 1,472 bytes received 127 bytes 1,066.00 bytes/sec total size is 3,101 speedup is 1.94
10. Limit the bandwidth
You can set the limitation of the bandwidth when running rsync command with '--bwlimit=<KB/s>' option.
Assuming that you want to limit the data transfer rate to 300KB/s, run the command:
$ rsync -avzh --bwlimit=300 /home/linoxide/coursera.png firstname.lastname@example.org: email@example.com's password: sending incremental file list coursera.png sent 2.93M bytes received 35 bytes 234.48K bytes/sec total size is 3.00M speedup is 1.02
11. Resume the transferring when failed in scp
Sometimes, you have to transfer a very large file via scp command but got error while copying and could not start transferring again using scp command due to the large file size and time consumption.
In this situation, you can use rsync command to start copying the file from where it got the error. For example, copying a large ISO file in home directory to remote host 'firstname.lastname@example.org'
$ scp /home/linoxide/ubuntu-18.04.4-live-server-amd64.iso email@example.com: firstname.lastname@example.org's password: ubuntu-18.04.4-live-server-amd64.iso 32% 278MB 86.7MB/s 00:06 ETA
The process has been killed by user. At that time, 32% of file was transferred. Now, you can continue the current work by running rsync:
$ rsync -P -avzh ubuntu-18.04.4-live-server-amd64.iso email@example.com: firstname.lastname@example.org's password: sending incremental file list ubuntu-18.04.4-live-server-amd64.iso 912.26M 100% 15.89MB/s 0:00:54 (xfr#1, to-chk=0/1) sent 541.25M bytes received 132.44K bytes 9.25M bytes/sec total size is 912.26M speedup is 1.69
12. Display progress while transferring data
If you want to see the transferred progress when running rsync command, you can use '--progress' option.
It will show the files and the time remaining to complete the copy.
$ rsync -avh --progress /home/linoxide/cloud/ email@example.com: firstname.lastname@example.org's password: sending incremental file list ./ computer-networking.pdf 3.03K 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=8/10) coursera.png 3.00M 100% 114.38MB/s 0:00:00 (xfr#2, to-chk=7/10) k8s.yaml 39 100% 1.52kB/s 0:00:00 (xfr#3, to-chk=6/10) kubernetes.yaml 34 100% 1.33kB/s 0:00:00 (xfr#4, to-chk=5/10) ubuntu-18.04.4-live-server-amd64.iso 912.26M 100% 89.60MB/s 0:00:09 (xfr#5, to-chk=4/10) document/ document/kubernetes.txt 0 100% 0.00kB/s 0:00:00 (xfr#6, to-chk=1/10) kuber/ kuber/test.txt 0 100% 0.00kB/s 0:00:00 (xfr#7, to-chk=0/10) sent 915.49M bytes received 164 bytes 79.61M bytes/sec total size is 915.26M speedup is 1.00
In this tutorial, we learned how to use rsync command with some examples for copying or syncing files/directories.
Thanks for reading and please leave your suggestion in the below comment section.