Resistance is futile
There’s a famous saying that “data that’s not backed up is data you’re prepared to lose.” I used Windows for a very long time, and managed to lose quite a bit of data back in the day because of either Windows Update bricking the system or just wanting to reinstall the OS (Windows has a habit of losing performance over time - easiest fix is a fresh install). I had been performing backups manually to an external hard disk, frequently forgot to backup something critical, and only had backups when I cared to make them (i.e. rarely). Fortunately, there’s a better way of doing things: automated backups to a remote server with setup-and-forget tools like rsync and borg. I haven’t lost data since.
Before we start
To use either of these tools, all you need is a UNIX system (Mac/Linux) and a server or storage device to back up to. There are no other requirements. If you don’t have a system of your own, I highly recommend Rsync.net. Rsync.net is a very cheap/reliable backup provider that simply gives you an SSH endpoint to dump your files in. Though plans vary, the price is about $1 per GB stored per year, which is quite affordable, and the service comes with free snapshots and support. If you choose this option, be aware that there’s also a secret pricing tier for borg users which gives heavily discounted plans that do not include support or snapshots (since borg does that for you).
If you’ll be backing up to a remote system, you’ll want to setup passwordless SSH before you start (all of the defaults are fine here, do not enter a passphrase for your key). This is more secure than using a password to connect, and it means that backups over SSH can be performed non-interactively.
ssh-keygen -t rsa # hit enter for all the prompts here, you typically do not want to set a passphrase ssh-copy-id email@example.com
Before you do anything else, make sure that connecting over SSH to your backup server no longer prompts you for a password.
Alternatively, if you are backing up to a network drive, make sure you know how to mount and unmount
the drive via the command line (typically via
umount). Your backups should never be
mounted to your computer unless you are backing things up!
(This reduces the risk of attackers or accidents breaking your precious backups.)
Once this is done, you should be set.
Simple backups with rsync
rsync is a very handy file-copying tool that performs easy, straightforwards backups.
Backups are unencrypted and unauthenticated, but it’s trivial to setup and restore from.
If all you want is an up-to-date backup when things go bad,
rsync is the tool for you.
To create a backup
This performs a very simple backup to any storage device. Files are copied as-is, and all attributes (ownership, permissions, modification times, etc.) are preserved. No authentication or encryption is performed, meaning that anyone could get at your files if they can access your storage media. Files that you delete are deleted from your backup, and only files modified since the last backup are uploaded.
To back up to a local disk or network drive:
rsync -az --delete /folder/to/back/up /destination/folder
To back up to a remote server:
Note that this command connects to the remote server over SSH, meaning that information is encrypted while being transferred to the remote server. The actual backups themselves, however, are unencrypted.
rsync -az --delete -e ssh /folder/to/back/up firstname.lastname@example.org:/destination/folder
To restore from a backup
To restore files from your remote backup, no special magic is required - just reverse
the source and destination folders in the above command. Alternatively, you can
use tools like
sftp to restore individual files.
rsync -az -e ssh email@example.com:/destination/folder /folder/to/restore/in
Pros and cons of rsync
rsync is useful for its simplicity. You get easy-to-perform backups for virtually no learning curve, and restoring files is a breeze (just copy them back!).
There are some big drawbacks to this method, however. rsync is very space inefficient - no compression or deduplication is performed. You only get access to a single backup as well. If you want multiple backups for multiple dates, you’ll need to manage these manually, and each extra backup will take up an equal amount of space (7 days’ of backups == 7 times the storage usage). Anyone with access to your storage media will also have access to your files.
In light of this info, rsync is a great tool for fast and dirty backups to local storage media, or when you are confident that your backup location is secure and cannot be accessed by anyone else. If you want multiple backups and access controls, you’ll need a different tool.
Secure backups with borg
Borg is a fantastic tool that covers the weaknesses of rsync without sacrificing much in terms of usability. In particular, you’ll be able to keep multiple backups, save space through deduplication and compression, and secure your data with either passwords or a keyfile.
Borg requires a little bit of additional setup before you can start using it.
Having borg installed on the remote server will speed things up.
This is already done for you if you use Rsync.net, although you should specify
the environment variable
BORG_REMOTE_PATH to use the most recent version of borg available:
# only for Rsync.net users export BORG_REMOTE_PATH=/usr/local/bin/borg1/borg1
Additionally, you will need to initialize your repository before you can use it. To create a new repository that’s password-protected, use the following:
# see "borg init --help" for more options like storage quotas, encryption options, etc. borg init -e repokey-blake2 firstname.lastname@example.org:/destination/folder
Creating a backup
For your first backup, you may wish to do it interactively so you can watch the progress
and verify that things work. The following creates a backup titled
borg create --progress --stats email@example.com:/destination/folder::backup-name /folder/to/back/up
For subsequent backups, you will likely want to do things non-interactively. You can use the
following to create an automatically-named backup (computer name + date).
Note that further commands assume you’ve set the
BORG_REPO environment variable
(specifying a default repository to back up to).
# specify a password for non-interactive use export BORG_PASSPHRASE='your repository password' # specify the default repository to use for backups export BORG_REPOfirstname.lastname@example.org:/destination/folder' borg create ::$(hostname)-$(date -I) /folder/to/back/up
Cleaning up old backups
Chances are, you will not want to keep every backup ever made. You might want to keep say only 7 days’ worth of daily backups, 8 weeks of weekly backups, and 12 months of monthly backups. To do so:
borg prune --keep-daily 7 --keep-weekly 8 --keep-monthly 12
To view a list of all backups:
borg list # follows is a list of backups, dates, and ids
To view all files within a backup (EXTREMELY VERBOSE, so output has been piped to
borg list ::backup-name | head
Restoring from a backup
Restoring files is quite easy, although they are extracted to the current working directory
(so if you backup up
/home/youruser/some-folder, expect it to recreate that directory structure
cd to the root directory).
To extract a single file:
cd /location/to/restore/to borg extract ::archive file/to/restore
To extract all files:
cd /location/to/restore/to borg extract ::archive
Automating your backups (and sample scripts!)
To run your backups automatically, you’ll want to create a script, and run it automatically through
crontab is normally a great way to do this (run a task at a specified time and day), it is not
very flexible - if you set it to perform backups at 3am, and you’re not logged onto your laptop at 3am,
the backup wont happen! Instead, we’ll create a script and put it in
/etc/cron.daily. Scripts here
are automatically run about 15 minutes after logging on to your computer.
Here are some sample scripts that you can use for either rsync or borg.
Installation is the same - just copy these to
You’ll note I’ve fully specified the path to programs here - this is a “best practice” when working with
scripts to be run under
rsync example script
#!/bin/bash # Backup a folder to a remote address using rsync. # Usage: backup-rsync.sh # To restore: rsync -az -e ssh email@example.com:backups/$(hostname)/folder /restore/point set -eu /usr/bin/ssh firstname.lastname@example.org mkdir -p backups/$(hostname) /usr/bin/rsync -az --delete -e ssh /folder/to/back/up email@example.com:backups/$(hostname)/
Borg example script
#!/bin/bash # Backup a folder to a remote address using borg. # Usage: backup-borg.sh # To restore: borg extract $BORG_REPO::computer-and-date set -eu export BORG_REPOfirstname.lastname@example.org:borg/repo/path' export BORG_PASSPHRASE='your password' export BORG_REMOTE_PATH=/path/to/remote/borg /usr/bin/borg create ::$(hostname)-$(date) /folder/to/back/up /usr/bin/borg prune ::$(hostname)-$(date) --keep-daily=14 --keep-monthly=6
Assuming you’ve added one of these scripts to your
all you have to do is wait. If you’ve added
BORG_REPO to your
you can check in and verify that your backups are working properly with
borg list (you should see a list of your current backups).