Backup

YouTrack initiated backups

YouTrack is configured to make a .tar.gz backup at 02:00 daily.

Storage

The backup archives are stored in /Users/tapadoo/Docker/youtrack/backups.

Redundant storage

A copy of these backups is also maintained on jessie in /Users/jessie/youtrack.backups. This folder is kept in sync with the slinky-dog folder, and it is updated once per day also. The sync script can be found at /Users/jessie/bin/sync-youtrack-backups.sh and it is controlled by launchd via /Users/jessie/Library/LaunchAgents/com.tapadoo.youtracksync.plist.

Offsite storage

In case of catastrophic failure we also main off site backups. As each backup archive is a large object (29GB at time of writing), writing these to cloud storage is an unnecessary used of resources. To handle the backups more efficiently we maintain a backup repository on slinky-dog, and then sync this repository with cloud storage.

Borg

The backup repository is created and managed by Borg:

BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.

The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.

Script

The backup script (heavily influenced by Automating Backups is located at /Users/tapadoo/Docker/youtrack_backup/backup.sh. It keeps:

• 7 daily
• 4 weekly and
• 6 monthly backups

The backups are stored in the borg_backup_youtrack repository located at /Users/tapadoo/Docker/youtrack_backup/borg_backup_youtrack.

Note: the password for this repository is stored in 1Password.

The borg executable is also located in the /Users/tapadoo/Docker/youtrack_backup directory in the borg-dir directory. The script is in the crontab for user tapadoo and it runs each night at midnight:

0 0  *  *  * /Users/tapadoo/Docker/youtrack_backup/backup.sh > /tmp/borg.log 2>&1

backup.sh is in the server-ops repository.

Cloudflare R2

The borg repository is backed up to the youtrack-backup bucket in Cloudflare’s R2 storage. rclone is used to copy the borg repository to the bucket. The rclone executable is also located in the /Users/tapadoo/Docker/youtrack_backup directory. Configuration for rclone is in the /Users/tapadoo/.config/rclone/rclone.conf file.

./rclone sync borg_backup_youtrack r2:youtrack-backup

sync_to_cloud.sh is in the server-ops repository. The script is in the crontab for user tapadoo and it runs each at 01:00 each day (line break added for readability):

0 1 * * * /Users/tapadoo/Docker/youtrack_backup/sync_to_cloud.sh > /tmp/r2_sync.log 2>&1

Restore

NOTE: Please use jessie when testing the restore procedure.

Restore YouTrack backups

See the JetBrains documentation on how to restore from a backup archive.

Offsite restore

Cloudflare R2

Download

Download the Borg repository from R2 (see server-ops for example rclone configuration file):

./rclone sync r2:youtrack-backup borg_backup_youtrack

Extract

Extract the desired archive. To get a list of the archive use the borg list command:

./borg-dir/borg.exe list borg_backup_youtrack

Example output:

Enter passphrase for key /Users/tapadoo/Docker/youtrack_backup/borg_backup_youtrack: 
youtrack-2023-11-01T17:33:46         Wed, 2023-11-01 17:33:46 [550a...df40]
youtrack-2023-11-01T18:41:01         Wed, 2023-11-01 18:41:01 [6719...446d]
youtrack-2023-11-02T00:00:02         Thu, 2023-11-02 00:00:02 [9ac8...6952]
youtrack-2023-11-03T00:00:01         Fri, 2023-11-03 00:00:02 [d162...7142]
youtrack-2023-11-04T00:00:00         Sat, 2023-11-04 00:00:01 [f881...9dbc]
youtrack-2023-11-05T00:00:01         Sun, 2023-11-05 00:00:01 [7e5f...94a8]
youtrack-2023-11-06T00:00:01         Mon, 2023-11-06 00:00:02 [695f....7a38]

Then extract:

./borg-dir/borg.exe extract borg_backup_youtrack::youtrack-2023-11-06T00:00:01

Now the backup is available in the youtrack directory:

jessie@jessie r2_test % ls -al youtrack 
total 0
drwx------   4 jessie  staff  128  6 Nov 12:20 .
drwxr-xr-x   4 jessie  staff  128  6 Nov 12:20 ..
drwxr-xr-x  10 jessie  staff  320  5 Jun 16:27 conf
drwxr-xr-x   6 jessie  staff  192  5 Jun 17:56 data

Start YouTrack

For Production

When launching YouTrack using Docker, mount these directories as volumes. Note the name below is for example purposes only. Port 8083 is important as this is the port using in the Cloudflare Zero Trust mapping to expose the service on https://bugs.tapadoo.com.

On slinky-dog:

docker run \
  --name youtrack_bak \
  -v $PWD/data:/opt/youtrack/data \
  -v $PWD/conf:/opt/youtrack/conf \
  -v $PWD/logs:/opt/youtrack/logs \
  -v $PWD/backups:/opt/youtrack/backups \
  -p 8083:8080 \
  jetbrains/youtrack:2020.6.4750

For Verification

On jessie:

docker run \
  --name youtrack_bak \
  -v $PWD/data:/opt/youtrack/data \
  -v $PWD/conf:/opt/youtrack/conf \
  -v $PWD/logs:/opt/youtrack/logs \
  -v $PWD/backups:/opt/youtrack/backups \
  -p 8084:8080 \
  jetbrains/youtrack:2020.6.4750

Visit https://ytb.tp.ie - this is a domain configured in Cloudflare Zero Trust to serve the youtrack_bak instance. The data can then be verified from here.

Restore verifications