Let's switchover es1023 to es1024 which is already running Buster and Mariadb 10.4.
The idea is to move writes to es4 for a few minutes and then do the switchover on es5 master.
Steps:
- Give weight 50 to es1024
- Disable alerts on es5 hosts
- switchover.py --timeout=15 --only-slave-move es1023.eqiad.wmnet es1024.eqiad.wmnet
- Disable puppet es1024 and es1023
- Merge puppet change to promote es1024 to master https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607236/
- Disable writes for es5 on MW: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/606663/
- !log Starting es5 failover from es1023 es1024
- ./switchover.py --skip-slave-move es1023 es1024
- Depool es1023 entirely
- First 10.4 master, so let's double check that all the slaves are well connected.
- Let es1023 replication stopped, as it will be running 10.1 and its master will run 10.4, let's reimage before starting replication.
- dbctl --scope eqiad section es5 set-master es1024
- Enable and run puppet at es1023 and es1024
- events_coredb_master.sql on the new master es1024
- events_coredb_slave.sql on the new slave es1023
- Revert the above patch to make es5 writable again.
- Change es5-master DNS https://gerrit.wikimedia.org/r/c/operations/dns/+/609899/
- Disable notifications on es1023
- Reimage es1023
- Slowly repool es1023 and remove weight from es1024
@jcrespo I would appreciate a review of the above simplified steps
Date & time: Tuesday 7th July at 05:00 AM UTC