Page MenuHomePhabricator

Upgrade s1 to Buster + MariaDB 10.4
Open, MediumPublic

Description

After the DC switch (scheduled for 14th Sept 2021)

  • Available backup source on the standby DC (db2141 - all sections have sources everywhere now)
  • Switchover backup generation standby DC (Ready to deploy: https://gerrit.wikimedia.org/r/c/operations/puppet/+/721285 )
  • Candidate master on the standby DC
  • Master on the standby DC
  • Candidate master on the primary DC
  • Available backup source on the primary DC (db1140 - all sections have sources everywhere now)
  • Switchover backup generation Primary DC (Ready to deploy: https://gerrit.wikimedia.org/r/c/operations/puppet/+/721286 )
  • Switchover on the primary DC to promote a Buster+10.4 host to master: TO-DO
  • Upgrade the old master and make it a candidate master, pool it
  • Cleanup (remove) old backup sources from both DCs

Please read the doc about procedure for more details.

Event Timeline

Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to Ready on the DBA board.

To be worked on after the DC switch

My petition for s1 and s8 is, within reason, if they could be relatively close one from the other? Close here doesn't have a strict meaning, just whatever period of time you could agree to.

I can try to get the stanby DCs to be done in parallel but the primary masters I would like to leave 2 weeks between master swaps

Yes, that's ok regarding backup reimages.

Change 721285 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Switch s1 backup generation from db2097 to db2141

https://gerrit.wikimedia.org/r/721285

Change 721286 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Switch s1 backup generation from db1139 to db1140

https://gerrit.wikimedia.org/r/721286

Change 721288 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Migrate s8 backups db2100 -> db2098; reimage dbprov2001

https://gerrit.wikimedia.org/r/721288

Change 721652 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db2103 to Buster

https://gerrit.wikimedia.org/r/721652

Change 721652 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db2103 to Buster

https://gerrit.wikimedia.org/r/721652

Change 723038 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2103: Disable notifications

https://gerrit.wikimedia.org/r/723038

Change 723038 merged by Marostegui:

[operations/puppet@production] db2103: Disable notifications

https://gerrit.wikimedia.org/r/723038

All s1 core codfw replica hosts upgraded to the minor 10.4.21
@jcrespo you might want to also upgrade the source backup (I can do so if you give me green light)

@jcrespo you might want to also upgrade the source backup (I can do so if you give me green light)

Done.