Page MenuHomePhabricator

Upgrade and restart m3 (phabricator) master (db1128)
Closed, ResolvedPublic

Description

In order to continue with T239791: DB: perform rolling restart of mariadb daemons to pick up CA changes we'd need to restart MySQL on m3 host (db1128).
As we have to restart MySQL, we will also upgrade it to the newer version.

We'd like if you can help us and set phabricator on read-only before we do the restart, as we normally do when there's a maintenance involving this master.

This restart shouldn't take more than a few seconds, the proxy will failover reads to the other replica, so reads shouldn't be affected that much, writes will obviously blocked for a few seconds.

We should probably email wikitech-l@ once we've picked up a date/time.

I was thinking about Wednesday 12th February at 06:00 AM UTC? Would that work for you @mmodell? I can do it earlier if needed, like 05:00 AM UTC if that works best for you.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2020, 9:05 AM
Marostegui triaged this task as Medium priority.Feb 7 2020, 9:05 AM
Marostegui moved this task from Triage to Next on the DBA board.

For reference, the similar maintenance performed at T244209 resulted in 74 seconds of downtime.

12th (the original date I suggested) has passed, any tentative date @mmodell you'd like to consider, there is no rush really, I just want to organize myself around this :-)
Thank you!

Hey @Marostegui, How about tomorrow? I can be around tomorrow, if you'd like. If you'd like to do it at your leisure, and for future reference, the command for read-only is as follows:

/srv/phab/phabricator/bin/config set cluster.read-only true

The command needs to run as root on phab1001.eqiad.wmnet

To disable read-only, obviously, just set the config to false:

/srv/phab/phabricator/bin/config set cluster.read-only false

I've also documented the above commands on wikitech, for posterity, at https://wikitech.wikimedia.org/wiki/Phabricator#read-only_mode_/_restarting_mariadb

Hey @Marostegui, How about tomorrow? I can be around tomorrow, if you'd like. If you'd like to do it at your leisure, and for future reference, the command for read-only is as follows:

/srv/phab/phabricator/bin/config set cluster.read-only true

The command needs to run as root on phab1001.eqiad.wmnet

To disable read-only, obviously, just set the config to false:

/srv/phab/phabricator/bin/config set cluster.read-only false

I've also documented the above commands on wikitech, for posterity, at https://wikitech.wikimedia.org/wiki/Phabricator#read-only_mode_/_restarting_mariadb

Tomorrow works for me - at what time? 06:00AM UTC?

Thanks for the commands, noted. If possible I rather have you around just in case, it feels safer :)

@Marostegui Thursday 6:00 AM works for me.

@Marostegui Thursday 6:00 AM works for me.

Excellent! See you tomorrow at 06:00AM UTC! Thank you!

Marostegui moved this task from Next to In progress on the DBA board.Feb 13 2020, 5:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-13T05:47:04Z] <marostegui> Silence m3 hosts for maintenance - T244566

Mentioned in SAL (#wikimedia-operations) [2020-02-13T05:53:17Z] <marostegui> Upgrade db1128 without restarting mysql - T244566

Mentioned in SAL (#wikimedia-operations) [2020-02-13T06:00:53Z] <marostegui> Start phabricator maintenance T244566

Marostegui closed this task as Resolved.Feb 13 2020, 6:10 AM

This was done successfully.
Downtime was 56 seconds:
06:01:19 - 06:02:15