Post mortem:

Over the past few days, the SQLite DB Sonarr uses was partially degrading for reasons we're not sure yet. Restoring should of been easy, and is a documented task with step-by-step instructions.

However, we discovered that the automated backups of Sonarr had been corrupted due to our script copying the ZIP file produced by Sonarr whilst it was still making the backup. Our script simply waited 5 seconds after sending the backup request to *arr.

We didn't detect this initially when testing and running a disaster recovery, as the backup was so small at the time, it was done in under 1 second, compared to the ≈15 seconds it takes on production now.

Due to this, the backups of Sonarr from the last 2 months were completely corrupted. Luckily, we also do snapshots of our server's disk on a weekly basis. One had been done recently, so we backed up our database containing user, group, and torrent data, and restored to that snapshot.

After restoring the main database, we now had to manually map ≈90 torrents that had an incorrect media ID compared to the proper Sonarr database. This didn't take long (10 minutes), as Sonarr could re-parse the title and assign the torrent the right ID.

This shouldn't of happened at all, and due to our ignorance, the site was degraded for the last 2 days.

We have now improved our backup script to ensure that *arr have finished backing up before copying to an off-site server. We hope to be able to serve the community without any more downtime.

All data has been restored and the site is fully online. Thank you for your patience. We are now putting in stronger backup measures to ensure that this won't happen again.

We have a backup which has 98% of the media in it. We are restoring and manually mapping the other 2% back. Site and tracker should be back up in around an hour.

Sonarr has corrupted and several backups are not valid. Please be patient whilst we restore the site.