Backing Up Your Server¶

Approximate Time: 30 minutes

Prerequisite: Configuring your Server

What to Back Up¶

It isn't important to back up the entire operating system.

If the server needs to be rebuilt, such as when changing providers or in the case of a hardware failure, you'd follow the steps in this guide again and reinstall things like fail2ban and docker as you did.

Better yet, you're using a declarative build and can reinstall everything with a single command.

Presumably your code is checked into git, so backing the code up isn't important either.

The only things that you need to back up are files created on the server (such as user uploads) and your databases.

Where to Store Backups¶

A commonly cited rule for backups is the 3-2-1 rule: 3 copies of your data, 2 different kinds of storage, 1 offsite.

For a Postgres database this might mean:

• Our live data in the database. Our first onsite copy.
• A second copy of the data locally, useful for local restores, our second onsite copy.
• A third copy stored on a different service, useful if we lose access to our server or even Hetzner account. We'll be using object storage for our offsite backups, which is distinct from the local drive based storage on the Hetzner instance.

Object Storage¶

Most cloud providers offer an "object storage" service. These services allow storing files in a redundant storage service where data loss is almost impossible due to comprehensive replication. They bill at around pennies per GB/mo and support a common interface based on Amazon S3's original implementation.

Backblaze B2¶

Backblaze B2 is object storage that's great for low cost backups.

Create a Bucket¶

Visit https://secure.backblaze.com/b2_buckets.htm

Give the bucket a unique name. Something like yoursite-backups should work.

Leave the other settings the same, most importantly 'Files in Bucket' are private.

Create a Key¶

Visit https://secure.backblaze.com/app_keys.htm

Set a descriptive name for the key, something like yoursite-backups-key. To increase security, you can restrict the key's access to the bucket you just created. Leave the rest of the options as their defaults.

This will give a keyID that looks like 00041e5476d1f9a000000000d.

And an applicationKey resembling K0001B0kq99Qa+aWT8rt1pc0W115lo0.

Copy your values down into a safe place. It is essential you keep them private.

restic¶

We are going to use a program called restic to manage taking and storing backups.

sudo apt install restic

Then we will create a random password:

openssl rand -base64 32 > ~/.restic-password
chmod 600 ~/.restic-password
cat ~/.restic-password

Create the Repository¶

restic needs a repository (an encrypted store inside your bucket) before the first backup. Create it once, using the same values the script will use:

export B2_ACCOUNT_ID="<keyID>"
export B2_ACCOUNT_KEY="<applicationKey>"
export RESTIC_REPOSITORY="b2:<bucket-name>:/<server-name>"
export RESTIC_PASSWORD_FILE="$HOME/.restic-password"

restic init

You should see a message like created restic repository at b2:yoursite-backups:/server-name.

To confirm it worked:

restic snapshots

An empty list with no errors means the repository exists and your credentials work.

Simple Backup Script¶

Create a directory named ~/backups and place this script at ~/compose/backup.sh:

#!/bin/sh
set -e
# CHANGE: set your account credentials, bucket name, and server name here
export B2_ACCOUNT_ID="<keyID>"
export B2_ACCOUNT_KEY="<applicationKey>"
export RESTIC_REPOSITORY="b2:<bucket-name>:/<server-name>"
export RESTIC_PASSWORD_FILE=$HOME/.restic-password

# back up our Miniflux database to a local file
cd ~/compose
docker compose exec -T db pg_dump -U miniflux -Fc miniflux > ~/backups/miniflux-$(date +%F).dump

# CHANGE: anything that you want to back up should wind up in this ~/backups directory. More databases? Locally created files?

# back up everything in ~/backups to our restic repository
restic backup ~/backups

# optional: don't keep backups forever
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

# clean up old backups from disk
find ~/backups -name '*.dump' -mtime +7 -delete

Once created run: chmod 700 ~/compose/backup.sh to make the script executable and only accessible to the current user.

Verifying Backups¶

It is important to verify the backups work as intended.

On the server run ~/compose/backup.sh and wait for it to complete successfully.

After it does, on your laptop, run:

export B2_ACCOUNT_ID="<keyID>"
export B2_ACCOUNT_KEY="<applicationKey>"
export RESTIC_REPOSITORY="b2:<bucket-name>:/<server-name>"

restic snapshots                      # list backups; newest should be recent
restic check                          # ensure repository structure is what you expect
restic check --read-data-subset=5%    # download and verify 5% of the data

A more complete check would be to restore the backup and inspect the data:

restic restore latest --target /tmp/restore-test
ls /tmp/restore-test        # confirm expected files are present

Scheduling Regular Backups¶

Now that we know our backups work, we can use cron to schedule regular backups:

crontab -e will open a special file that tells cron what to run when. If there are existing lines leave them alone, add the new line:

0 3 * * * ~/compose/backup.sh >> ~/backups.log 2>&1

This will tell cron to run your backup script every day at 3am (in the server's timezone), appending all output to the file ~/backups.log. Feel free to change the time or log location for your own purposes.

Confirming cron Is Running¶

After a day or two, revisit the steps to verify your backups to make sure you're seeing continuous updates.

If you aren't seeing what you expect, check journalctl -u cron --since today to see if the job itself ran.

You can also check ~/backups.log to see if any error messages print. If backups are running on their own, you're done for now. It is probably a good idea to set a reminder to check a backup every month or two.