Media backups has a lot of moving parts, which can fail at any moment (lot of distribution over the network, several shards and many filesystems, potential consistency issues between metadata and phisical data on disk, suffer lag, making recovery impossible, depends on several separate databases for metadata, clients not having its disk filled in, storage not being filled in, swift download being available (including enough access grants), mediawiki api being available, wikimedia data model being static, etc.)

We need to make sure recovery is possible at all times (or more preciselly, above a certain SLA percentage of time) with enough freshness so we are ready in the event of an unplanned data loss, and we should start with a few key alerts:

• The storage service is up (versitygw) on all active storage hosts
• The demons that send the files to backup are active and do not have fatal errors (compared to one time errors for a single file)
• The files backed up are updated up to a timestime, not too behind real time on Mediawiki
• TLS files for storage daemons and monitoring are valid: T430023 T295594