Due to the failure of cloudvirt1018 last month both of our database instances at the time suffered disk corruption and the work to deal get stuff working again happened in T216404: deployment-db03.deployment-prep.eqiad.wmflabs instance can not start and T216067: Recover from corrupted beta MySQL slave (deployment-db04).
The current situation looking at db-labs.php is that:
- deployment-db03 is unused (theoretically, I haven't dug into what else could be talking to this beyond MW so don't rely on it)
- deployment-db04 is the master
- deployment-db05 is a fresh slave which we should keep and possibly make the new master
When investigating puppet failures today I noticed that deployment-db04 had all sorts of nonsense like puppet function files overwritten with apt data and junk. I do not trust it and we should delete it. db03 is in a similar position though it's at least seemingly unused.
Added to which they're jessie and T218729: Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster.