tools-db-1.tools.eqiad1.wikimedia.cloud will become the new ToolsDB primary, replacing clouddb1001.clouddb-services.eqiad1.wikimedia.cloud.
We plan to make the switch on Apr, 6 2023 at 17:00 UTC.
The following checklist is a work-in-progress and contains all the steps that need to be performed. We can use the comments below to discuss.
- Make clouddb1001 read-only
SET SESSION sql_log_bin=0; FLUSH TABLES WITH READ LOCK; SET GLOBAL read_only = 1;
- Point the DNS tools.db.svc.eqiad.wmflabs to tools-db-1 (via wikireplica_dns.yaml)
$ # Merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/906053 $ ssh cloudcontrol1005.wikimedia.org $ source <(sudo cat ~root/novaenv.sh) $ wmcs-wikireplica-dns
- Check that tools are starting to connect to the new host
- Promote tools-db-1 to primary, by stopping replication and enabling read-write mode (official docs)
STOP ALL SLAVES; RESET SLAVE ALL; SHOW MASTER STATUS; SET GLOBAL read_only = 0;
- move floating ip (185.15.56.15) to tools-db-1 OR update the floating IP in puppet (used by maintain-dbusers) (Moving IPs between projects isn't supported, so my current plan is to release the ip from clouddb-services and then allocate a whole lot of new ones in 'tools' and hope to get lucky.)
- Stop mariadb on clouddb1001 to close all remaining connections to the old db
- Change puppet class in horizon from role::wmcs::db::toolsdb_secondary to role::wmcs::db::toolsdb_primary (this doesn't change much as the two roles are almost identical and should probably be merged into one).