Rsync(Remote Sync) is one of the most efficient and secure data copying tool in Linux and Unix world. Rather than just copying, it can be called as synchronization tool. Whenever rsync is used to copy the files, it does it like a copy for the first time, but during its successive iterations, it will synchronize the files by just copying the delta change between them by using the remote-update protocol. And this synchronization can be either between the files on the local system or on the local system and remote system. Mainly between the local and the remote, as rsync generally termed for “Remote Synchronization”.
NOTE: The rsync command cannot be used to copy the files between the two remote systems. Either you can copy data from local machine to remote machine or remote machine to a local one.
In this post, we will see some examples of rsync along with some rarely known options that work wonders in the Linux data administration.
Linux/Unix command syntax for ‘rsync’
rsync [options] [SRC] [DEST]
Rsync can be used either in the push mode or a pull mode when the remote system is in scope. The basic syntax would remain same:
For pushing the files to a remote system using Rsync
rsync [options] [SRC] user@host:[DEST]
Pulling the files from the remote system using rsync
rsync [options] user@host:[SRC] [DEST]
Let’s jump onto some basic use cases for rsync :
Example 1: We can use rsync as a replacement to cp command to copy files between the directories on the same system. Suppose if we want to copy all the files in /etc whose name contain serv use below command.
root@sujitkumar:~# pwd /root root@sujitkumar:~# ls root@sujitkumar:~# rsync /etc/*serv* ~ skipping directory insserv skipping directory insserv.conf.d root@sujitkumar:~# ls insserv.conf services root@sujitkumar:~#
Example 2: Usually we want to copy and synchronize the directories across the systems.
rsync /path/*file* user@host:/home/
Example:
root@sujitkumar:~# rsync -rv /etc/default sujit@10.10.1.125:/home/sujit/ sujit@10.10.1.125's password: sending incremental file list default/ default/acpid default/bootlogd default/console-setup default/crda default/cron default/dbus default/devpts default/ntfs-3g default/ntpdate default/puppetmaster sent 19620 bytes received 529 bytes 3663.45 bytes/sec total size is 17982 speedup is 0.89 root@sujitkumar:~#
The ‘r’ option copies the ‘foo’ directory in a recursive manner into ‘/home/user’ directory.
I will use the rsync between the local and the remote system, as that is required in the majority of cases. And since we are copying the directories, we need to do it recursively. Verbose with any command informs about the command execution, hence we can use ‘-v’ option for detailed output flow.
Note: The permissions, ownership, and groups are not retained for the rsynced files and directories on the remote system. Files owned by root will be owned by ‘sujit’ with default permissions for the discussed example.
Example 3: Archive files once copied.
root@sujitkumar:~# rsync -av /etc/default/ sujit@10.10.1.125:/home/sujit/
The ‘a’ option is for archive, this will synchronize the contents of ‘default’ directory recursively, preserving symbolic links, special and device files, modification times, group, owner, and permissions.
Note:1 Here I have one interesting point to make, trailing slash ‘/’ after ‘default’, synchronizes only the contents of default directory, without it the directory ‘default’ along with its contents would be synchronized on the remote system.
Note:2 The owner and groups are retained in the remote user is a super-user, and the user and group exist there.
Example 4: Sometimes you want to do a dry-run before you actually run the rsync, so that you know what data will be transferred during the execution, so we use ‘n’ and ‘v’ options.
Example:
root@sujitkumar:~# rsync -nv /etc/default sujit@10.10.1.125:/home/sujit/ sujit@10.10.1.125's password: skipping directory default sent 8 bytes received 12 bytes 4.44 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) root@sujitkumar:~#
‘n’ option is used to do a dry run, in order to understand what the dry run does, we use the ‘v’ – verbose option that provides the information of what the command executes.
Example 5: For tracking the progress during the rsync data transfer, we use ‘p’ option and can be used as below:
root@sujitkumar:~# rsync -avP /etc/default sujit@10.10.1.125:/home/sujit/ sujit@10.10.1.125's password: sending incremental file list default/ default/acpid 346 100% 0.00kB/s 0:00:00 (xfer#1, to-check=26/28) default/apache2 637 100% 622.07kB/s 0:00:00 (xfer#2, to-check=25/28) default/apport 149 100% 145.51kB/s 0:00:00 (xfer#3, to-check=24/28) default/bootlogd 47 100% 22.95kB/s 0:00:00 (xfer#4, to-check=23/28) default/console-setup 2025 100% 988.77kB/s 0:00:00 (xfer#5, to-check=22/28) default/crda 549 100% 268.07kB/s 0:00:00 (xfer#6, to-check=21/28) default/cron 183 100% 89.36kB/s 0:00:00 (xfer#7, to-check=20/28) ............... default/ssh 133 100% 25.98kB/s 0:00:00 (xfer#23, to-check=4/28) default/tomcat7 1960 100% 382.81kB/s 0:00:00 (xfer#24, to-check=3/28) default/ufw 2072 100% 404.69kB/s 0:00:00 (xfer#25, to-check=2/28) default/useradd 1118 100% 218.36kB/s 0:00:00 (xfer#26, to-check=1/28) default/whoopsie 31 100% 5.05kB/s 0:00:00 (xfer#27, to-check=0/28) sent 19622 bytes received 529 bytes 3663.82 bytes/sec total size is 17982 speedup is 0.89 root@sujitkumar:~#
Example 6: Sometimes along with archive, compression is expected for faster data transfers. For compression based transfers we use ‘-z’ option along with archive option for faster operation along with the preservation of the directory/files attributes on the source.
root@sujitkumar:~# rsync -azvP /etc/default sujit@10.10.1.125:/home/sujit/ sujit@10.10.1.125's password: sending incremental file list default/ default/acpid 346 100% 0.00kB/s 0:00:00 (xfer#1, to-check=26/28) default/apache2 637 100% 622.07kB/s 0:00:00 (xfer#2, to-check=25/28) default/apport 149 100% 72.75kB/s 0:00:00 (xfer#3, to-check=24/28) default/bootlogd 47 100% 22.95kB/s 0:00:00 (xfer#4, to-check=23/28) default/console-setup 2025 100% 988.77kB/s 0:00:00 (xfer#5, to-check=22/28) default/crda 549 100% 178.71kB/s 0:00:00 (xfer#6, to-check=21/28) default/cron 183 100% 59.57kB/s 0:00:00 (xfer#7, to-check=20/28) default/dbus 297 100% 96.68kB/s 0:00:00 (xfer#8, to-check=19/28) default/devpts 92 100% 29.95kB/s 0:00:00 (xfer#9, to-check=18/28) default/grub 1225 100% 398.76kB/s 0:00:00 (xfer#10, to-check=17/28) default/halt 86 100% 27.99kB/s 0:00:00 (xfer#11, to-check=16/28) default/irqbalance 126 100% 41.02kB/s 0:00:00 (xfer#12, to-check=15/28) default/keyboard 559 100% 181.97kB/s 0:00:00 (xfer#13, to-check=14/28) default/locale 19 100% 4.64kB/s 0:00:00 (xfer#14, to-check=13/28) default/nss 1756 100% 428.71kB/s 0:00:00 (xfer#15, to-check=12/28) default/ntfs-3g default/ufw 2072 100% 404.69kB/s 0:00:00 (xfer#25, to-check=2/28) default/useradd 1118 100% 218.36kB/s 0:00:00 (xfer#26, to-check=1/28) default/whoopsie 31 100% 6.05kB/s 0:00:00 (xfer#27, to-check=0/28) sent 11349 bytes received 529 bytes 2159.64 bytes/sec total size is 17982 speedup is 1.51 root@sujitkumar:~#
Note: You can just verify the difference in the data bytes and the speedup in example 5 and example 6, which is because of the compression (-z) based data transfer.
Example 7: One of the major use cases of rsync is its support to include and exclude the specific files (also supports the regex) for the data transfer.
Consider that we are interested in transferring only the files with a pattern that starts with ‘puppet’, we can achieve this as below :
root@sujitkumar:~# rsync -avzP --include='puppet*' --exclude='*' /etc/default/ sujit@10.10.1.125:/home/sujit sujit@10.10.1.125's password: sending an incremental file list puppetmaster 418 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1/3) puppetqd 875 100% 854.49kB/s 0:00:00 (xfer#2, to-check=0/3) sent 857 bytes received 50 bytes 201.56 bytes/sec total size is 1293 speedup is 1.43 root@sujitkumar:~#
We can also manage all the files to be included or excluded for the data transfer in a file and mention it using ‘–exclude-from’ and ‘–include-from’ options.
Example 8: In case we want only the new files to be synchronized between the directories but the old files already transferred, need not be touched on the destination, then we can use the rsync with ‘u’ option:
root@sujitkumar:~# rsync -avzPu /etc/default/ sujit@10.10.1.125:/home/sujit
Here rsync would transfer only the new files from the source to the destination without replacing the modified files of the destination server, which is usually an expected case on the live application servers.
Example 9: Rsync for deletion of the files on source or destination.
‘-delete’ option is used with rsync to delete the files/directory that do not exist on the ‘src’ but already exist on the destination so that src and destination are synchronized.
root@sujitkumar:~# rsync -avzP --delete /etc/default/ sujit@10.10.1.125:/home/sujit sujit@10.10.1.125's password: sending an incremental file list deleting file4 deleting file3 ./ puppetmaster 418 100% 0.00kB/s 0:00:00 (xfer#1, to-check=9/28) sent 729 bytes received 40 bytes 53.03 bytes/sec total size is 17982 speedup is 23.38
Example 10: Other use case is that we might also come up with a requirement of deleting the source directory after its synchronization with the target. This can be handled by –remove-source-files option:
root@sujitkumar:~# rsync --remove-source-files -avzP --delete /etc/default/
Example 11: Though rsync transfers the only the incremental data, if bandwidth is not the matter of concern for the data transfer, the ‘-W’ option can be used to transfer the complete data from source to destination. This would generally speeds-up the rsync process as there is no “checksum’ comparison required for the same.
root@sujitkumar:~# rsync -avzP /etc/default/ sujit@10.10.1.125:/home/sujit
Example 12: All the examples above are highlighted with the push operation and using the remote ssh shell, rsync can be used in the pull operation from the destination server even(shown below):
sujit@sujitremote:~# rsync -avzP root@sujitkumar:/etc/default/ /home/sujit
Last but not the least, if the only src is mentioned without the destination in the rsync command, then it lists all the files/directories similar to the one we obtain from “ls -l” command.
Trying rsync in&out with its options can make wonders in your data transfer speed, accuracy, and efficiency.
author_sujit
Latest posts by author_sujit (see all)
- 12 tar command examples in Linux/Unix - September 26, 2017
- 9 bzip and bunzip command examples in Linux/Unix - September 22, 2017
- 12 rsync commands examples in Linux/Unix - September 22, 2017
