Page MenuHomePhabricator

Switchover m3 (phabricator) master db1072 to db1128
Closed, ResolvedPublic

Description

We will be switching over our primary database master for m3 (phabricator) which is db1072 to db1128.

db1072 is and old and out of warranty host that needs to be decommissioned.

When: Thursday 25 July - 05:30 - 06:00 AM UTC.
Impact: Writes will be blocked. Reads will remain unaffected.

The procedure will be as follows:

  • @Marostegui will do all the pre-switchover steps (including topology changes)
  • @mmodell will set up read-only for phabricator
  • @Marostegui will execute the switchover to promote db1128 to master and db1072 to slave
  • @Marostegui will reload haproxies to make db1128 primary on the proxy configuration
  • If everything looks good:
  • @mmodell to set up phabricator as read-write again.

We don't expect to use the full 30 minutes read-only window if everything goes fine.
The database switchover+proxy reload part should only take a few seconds. So I would expect just a few minutes of read-only as we need to get also phabricator read-only ON and OFF from the application layer.

Details

Related Gerrit Patches:
operations/puppet : productionmariadb: Promote db1128 as master for m3

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 17 2019, 5:59 AM
Marostegui triaged this task as Normal priority.Jul 17 2019, 6:00 AM
Marostegui moved this task from Triage to In progress on the DBA board.Jul 17 2019, 8:39 AM

Change 523941 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1128 as master for m3

https://gerrit.wikimedia.org/r/523941

Mentioned in SAL (#wikimedia-operations) [2019-07-25T04:51:54Z] <marostegui> Start pre-failover steps on m3 T228243

Change 523941 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1128 as master for m3

https://gerrit.wikimedia.org/r/523941

Marostegui closed this task as Resolved.Jul 25 2019, 5:41 AM

This has been done.
Phabricator read only start: 05:30:44
Phabricator read only stop: 05:31:37

Total read-only time: 53 seconds

Thanks @mmodell for being around to handle the phabricator side of things.