Performing backups and replicating files and folders securely is of vital importance when you need to recover from a disaster scenario such as a system crash, system wipe, a data loss incident or even a complete cloud compromise. The default Linux binaries already provide solid tooling to backup and sync files and directories.
PGP (Pretty Good Privacy) or OpenSSL can be used for encrypting files, but are rather inconvenient and complicated to use for file encryption. Other (paid) closed source backup solutions might not provide the proper security code review. This is why we use age (Actually Good Encryption) for asymmetric file encryption. age is like PGP, but with modernized cryptography, stripped out bloated features and supports authenticated encryption (AEAD). age got officially released as open source software in March 2021 by Google’s researcher Filippo Valsorda et. al.
The backup method described in this blog post adheres where possible to a zero-trust backup architecture, where we assume a Azure or AWS VPC (Virtual Private Cloud) is compromised via hostile actions, vulnerabilities or misconfiguration, and all S3 buckets/backups are now leaked on the dark web. This means we need to have strong authenticated encryption for certain receivers, namely only the backup entities.
Installing age and checking prerequisites
age is not yet available (march 2021) in common Linux apt repositories. Therefore we download age and place the binary in the /usr/local/bin/ folder so we can call age at all times. We used Ubuntu 20.04, Debian 10 and RHEL 8 for this blog post, although this backup method should work on almost all current Linux and BSD distributions.
backup-vps-01$ wget https://github.com/FiloSottile/age/releases/download/v1.0.0-rc.1/age-v1.0.0-rc.1-linux-amd64.tar.gz
backup-vps-01$ tar -xvf age-v1.0.0-rc.1-linux-amd64.tar.gz
backup-vps-01$ cd age
backup-vps-01$ cp age* /usr/local/bin/
backup-vps-01$ age
Now we check if the other system binaries are present:
backup-vps-01$ which ssh ssh-keygen scp tar cron
/usr/bin/ssh
/usr/bin/ssh-keygen
/usr/bin/scp
/bin/tar
/usr/sbin/cron
Configuring encrypted backup
Next we encrypt files at rest with age using a SSH public private key-pair. SSH (Secure SHell) is normally used to authenticate to servers, containers and workstations. However, we leverage a SSH key-pair to encrypt and decrypt files together with a passphrase. SSH cryptography encrypts and decrypts with two different keys.
The ssh-keygen binary generates an ed25519 SSH key-pair. The public key (id_ed25519.pub) is available for any system/user, while the private key (id_ed25519) should not be shared at all. Backup data is encrypted with the public key (id_ed25519.pub) and can only be decrypted with the private key (id_ed25519). You can read more here on SSH public key hardening. The backup process is shown in figure 1.
Figure 1: Encrypted backups process
Step 1: Generate an ed25519 SSH key-pair on the backup server with a strong passphrase-protected string and store this passphrase in a password manager:
backup-vps-01$ ssh-keygen -o -a 256 -t ed25519 -C "$(hostname)-age-backup-key-$(date +'%d-%m-%Y')"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/$USER/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /$USER/.ssh/id_ed25519
Your public key has been saved in /$USER/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:WpDO66RaSrwC5v5epgB2OVtyusIi3heEttNhjwW/9ew backup-vps-01-age-backup-key-25-03-2021
The key's randomart image is:
+--[ED25519 256]--+
| |
| . . |
| . = |
| o.* + . |
|...==oB S o |
|+o.oB+ * o |
|=.oooo= . |
|+=o+== E |
|=+BBo . |
+----[SHA256]-----+
backup-vps-01$ file /$USER/.ssh/id_ed25519.pub #OpenSSH ed25519 public key
backup-vps-01$ file /$USER/.ssh/id_ed25519 #OpenSSH ed25519 private key
Step 2: Sent the backup server public key over to the application server:
backup-vps-01$ cat /$USER/.ssh/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIEalg6zU9VDdL90TscKlPSCNjptsDRTgi7+0cY2wFlz nl-p-backup01-age-backup-key-25-03-2021
app-vps-01$ vi /$USER/home/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIEalg6zU9VDdL90TscKlPSCNjptsDRTgi7+0cY2wFlz nl-p-backup01-age-backup-key-25-03-2021
Step 3: Compress the backup data and encrypt with the SSH public key of the backup server:
app-vps-01$ tar -zcv /dir/you/want/to/backup/ | age -e -R /$USER/home/id_ed25519.pub > /$USER/home/$(hostname)-backup-$(date +'%d-%m-%Y').tar.gz.age
Step 4: Sent over the encrypted backup file to the backup server with scp (secure SSH copy), rsync or use your API to copy to a S3 bucket or any other cloud storage:
app-vps-01$ scp /$USER/home/app-vps-01-backup-25-03-2021.tar.gz.age backup-user@backup-vps-01:/destination/dir/
Step 5: Test decryption with the private key of the backup server and uncompress the archive file:
backup-vps-01$ age -d -i /$USER/.ssh/id_ed25519 app-vps-01-backup-25-03-2021.tar.gz.age > app-vps-01-backup-25-03-2021.tar.gz
backup-vps-01$ tar -xf app-vps-01-backup-25-03-2021.tar.gz
Congratulations. You now successfully compressed, encrypted, decrypted and decompressed backup data.
Considerations
Note that SSH keys could be revoked when the public/private keys are re-used for system authentication. Re-using an existing SSH key-pair for encryption is therefore not recommended in certain scenarios. SSH keys also embed a public key tag in the encrypted *.age file. This makes an encrypted backup file trackable to the respective SSH public key. This is also not necessarily a security risk. You can use the age-keygen binary instead to generate a ed25519 key-pair, or create a separate SSH key-pair for system authentication and backup encryption.
You can use cron, CI/CD pipelines or Ansible to automate this backup process on a daily/weekly/monthly basis. crontab can either directly call the one-liners to compress, encrypt and transfer the backup file(s), or call a Python/Shell script which does the same but with rsyslog logging and error handling. Read this blog post for more information.
Storing private keys on a server in plaintext is bad security practice. Do not leave the private key on the backup server. Only place id_ed25519 on the backup server when you need to decrypt a backup file. You can also leverage a Hardware Security Module (HSM) as an alternative.
Lastly, encrypted *.age files are not signed. Keep this in mind.
