Backup

YouTrack initiated backups

YouTrack is configured to make a .tar.gz backup at 03:00 UTC daily.

Storage

The backup archives are stored in our Hetzner volume mounted at:

/mnt/youtrack_storage/youtrack_backups.

Redundant incremental storage

In case of catastrophic failure we also maintain incremental off site backups. As each backup archive is a large object (29GB at time of writing), writing these to cloud storage is an unnecessary used of resources. To handle the backups more efficiently we maintain a backup repository, and sync this repository to a Hetzner Storage Box: YouTrack Borg.

Borg

The backup repository is created and managed by Borg:

BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.

The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.

Script

The backup script (heavily influenced by Automating Backups is located at /mnt/youtrack_storage/bin/backup.sh. It keeps:

• 7 daily
• 4 weekly and
• 6 monthly backups

The backups are stored in /home/youtrack_borg_backup in the Hetzner Storage Box.

Note: the password for this repository is stored in 1Password.

The borg executable is also located in /mnt/youtrack_storage/bin in the borg-dir directory. The script is in the crontab for user root and it runs each night at 3AM:

0 3  *  *  * /mnt/youtrack_storage/bin/backup.sh > /tmp/borg.log 2>&1

backup.sh is in the server-ops repository.

Cloudflare R2

The borg repository is backed up to the youtrack-backup bucket in Cloudflare’s R2 storage. rclone is used to copy the borg repository to the bucket. The rclone executable is also located in the /mnt/youtrack_storage/bin directory. Configuration for rclone is in the /root/.config/rclone/rclone.conf file.

./rclone sync borg_backup_youtrack r2:youtrack-backup

sync_to_cloud.sh is in the server-ops repository. The script is in the crontab for user root and it runs each at 03:15 each day (line break added for readability):

15 3 * * * /mnt/youtrack_storage/bin/sync_to_r2.sh > /tmp/r2_sync.log 2>&1

Restore

Restore YouTrack backups

See the JetBrains documentation on how to restore from a backup archive.

Offsite restore

Cloudflare R2

Download

Download the Borg repository from R2 (see server-ops for example rclone configuration file):

./rclone sync r2:youtrack-backup borg_backup_youtrack

Extract

Extract the desired archive. To get a list of the archive use the borg list command:

./borg-dir/borg.exe list borg_backup_youtrack

Example output:

Enter passphrase for key /Users/tapadoo/Docker/youtrack_backup/borg_backup_youtrack: 
youtrack-2023-11-01T17:33:46         Wed, 2023-11-01 17:33:46 [550a...df40]
youtrack-2023-11-01T18:41:01         Wed, 2023-11-01 18:41:01 [6719...446d]
youtrack-2023-11-02T00:00:02         Thu, 2023-11-02 00:00:02 [9ac8...6952]
youtrack-2023-11-03T00:00:01         Fri, 2023-11-03 00:00:02 [d162...7142]
youtrack-2023-11-04T00:00:00         Sat, 2023-11-04 00:00:01 [f881...9dbc]
youtrack-2023-11-05T00:00:01         Sun, 2023-11-05 00:00:01 [7e5f...94a8]
youtrack-2023-11-06T00:00:01         Mon, 2023-11-06 00:00:02 [695f....7a38]

Then extract:

./borg-dir/borg.exe extract borg_backup_youtrack::youtrack-2023-11-06T00:00:01

Now the backup is available in the youtrack directory:

jessie@jessie r2_test % ls -al youtrack 
total 0
drwx------   4 jessie  staff  128  6 Nov 12:20 .
drwxr-xr-x   4 jessie  staff  128  6 Nov 12:20 ..
drwxr-xr-x  10 jessie  staff  320  5 Jun 16:27 conf
drwxr-xr-x   6 jessie  staff  192  5 Jun 17:56 data

Start YouTrack

For Production

When launching YouTrack using Docker, mount these directories as volumes. Note the name below is for example purposes only. Port 8083 is important as this is the port using in the Cloudflare Zero Trust mapping to expose the service on https://bugs.tapadoo.com.

On Hetzner:

docker run \
  --name youtrack_bak \
  -v $PWD/data:/opt/youtrack/data \
  -v $PWD/conf:/opt/youtrack/conf \
  -v $PWD/logs:/opt/youtrack/logs \
  -v $PWD/backups:/opt/youtrack/backups \
  -p 8083:8080 \
  jetbrains/youtrack:2020.6.4750

For Verification

On jessie:

docker run \
  --name youtrack_bak \
  -v $PWD/data:/opt/youtrack/data \
  -v $PWD/conf:/opt/youtrack/conf \
  -v $PWD/logs:/opt/youtrack/logs \
  -v $PWD/backups:/opt/youtrack/backups \
  -p 8084:8080 \
  jetbrains/youtrack:2020.6.4750

Visit https://ytb.tp.ie - this is a domain configured in Cloudflare Zero Trust to serve the youtrack_bak instance. The data can then be verified from here.

Restore verifications