rsync

rsync (or remote sync) is a command line utility for transferring and synchronizing files between any given “source” and “destination” pair. It is extremely popular for use in backup utilities, at it can synchronize backups between not only local filesystems, but also over the internet via SSH.

While you can technically use the cp -R command to make quick and dirty backups, rsync offers far more flexibility for many common real-world scenarios.

If you are already familiar with using Wget to make mirrors of entire websites, then use of rsync should come to you pretty naturally - it just operates over local filesystems & SSH instead of HTTP/HTTPS/FTP.

Using rsync may seem obtuse to new users at first. However, once you know how to structure and alias your commands, it is one of the absolute fastest & reliable ways to run backup systems:

rsync –h and man grep will tell you all the flags which are available to use. We will only detail some of the most important ones below:

To have rsync copy all files from one directory to another, use this command:

$ rsync -av /path/to/source/dir/ /path/to/destination/dir/

The -a flag tells rsync to not only recursively go into subdirectories, but it also ensures all attributes like timestamps & permissions are preserved. -v is for verbose output, so you can see what rsync is doing.

rsync supports the use of SSH for transferring backups over network connections, using the -e flag.

To back up files from your local shell to a remote machine via SSH:

$ rsync -avz -e ssh /source/directory/ user@remote:/destination/directory/

Here we are also using the -z flag, which compresses the data to speed up transfer.

To back up files from a remote machine to your local shell via SSH:

$ rsync -avz -e ssh user@remote:/source/directory/ /destination/directory/

In the case of both of these commands, “local shell” just means whatever terminal you are running rsync from. And no, you cannot use rsync between a remote source and remote destination at the same time:

$ rsync -avz -e ssh user@remote1:/source/directory/ user@remote2:/destination/directory/
The source and destination cannot both be remote.

So if you want to sync between two remote machines, you will have to first SSH into one of them and run rsync from there “locally” with the other machine as the remote.

If you are regularly backing up the same folders, drive, server, etc, then rsync needs to be tweaked further with two more flags, or over time your backups are not going to look like the folders you were backing up:

$ rsync -avu --delete /path/to/source/dir/ /path/to/destination/dir/

With the -u (or –update) flag, we can tell rsync to only copy files from the source that do not exist in the destination, or are newer than the same files in the destination, and skip all the others. With the –delete flag, we can tell rsync to delete files in the destination that no longer exist in the source.

This is the form of rsync that you want to use in automated backup scripts.

If you are looking for an even more failsafe way to ensure the quality of backups, use the -c flag to have rsync compare checksums of every conflicting file transfer instead of modification times. This method is a lot slower, but far more accurate when left on autopilot.

When rsync has one of its backup runs interrupted, it just stops. Any files that were finished, you get to keep. Whatever file it was on last, is lost. Just run rsync again in incremental mode (see above) and it will start over on whatever file it was processing when it was interrupted.

However, if you're dealing with very large files, you do not want to have to start over. Use the -P flag if you want to keep partially-downloaded files, so you can resume transfers mid-file instead of between files.

If there are files/folders that you wish to have rsync skip over, use the –exclude flag like this:

$ rsync -av --exclude 'filename' --exclude 'path/' /path/to/source/dir/ /path/to/destination/dir/

If you have a lot of exclusions, you can list them all out one-per-line in a text file, and pass it to rsync with –exclude-from like this:

$ rsync -av --exclude-from 'exclusions.txt' /path/to/source/dir/ /path/to/destination/dir/

You can even exclude things like specific filetypes:

$ rsync -av --exclude='*.ext' /path/to/source/dir/ /path/to/destination/dir/

There are more flags you can combine with these basic rsync commands to fine-tune how your backups are executed. Play around with them on some demo directories to see how they affect your files!

• Use the -n or –dry-run flag in combination with -v to perform a simulated transfer.
• Use –progress to see real-time transfer status.
• Use –log-file=transfer.log to generate a log file of the transfer.
• Use –remove-source-files to delete the source directory once the transfer is complete.

rsync does not have a built-in scheduling service. The easiest way to set your rsync-powered backups on autopilot, is with the use of crontab.