Page MenuHomePhabricator

Restart m3 (phabricator) database master db1132
Closed, ResolvedPublic

Description

In order to be able to enable report_host variable on db1132 we'd need to issue a mariadb daemon restart for db1132.
We expect this to take around 1 minute.

When: Wed 27th at 06:00 AM UTC
Impact: Phabricator will be on read-only for around 1 minute.

@mmodell if possible, would be around for this? Let me know when would be a good time for you? I start working pretty early in the UTC morning (around 6AM UTC) so any time around that or later (whatever fits best your schedule) would work.

Event Timeline

Marostegui moved this task from Triage to Ready on the DBA board.
Marostegui updated the task description. (Show Details)

@Marostegui I can do 06:00 UTC, I don't usually go to bed until a bit later than that. I guess any day is fine, I think 06:00-07:00 UTC is actually the best time that would overlap both of our awake times.

Thanks @mmodell - going to send a calendar invite for 06:00 AM UTC!

Window reserved on the deployments page

Procedure:

Pre restart

  • Silence m3 hosts
  • buffer pool dump + disablement in advance to make the restart faster

Restart

  • !log m3 master restart, phabricator will go on read only - T272596
  • set phabricator in RO
  • db1132: restart mysql
  • verify report_host is enabled
  • verify read_only is OFF
  • Once mysql is back: reload haproxy on dbproxy1020 (active) and dbproxy1016 (passive)
  • set phabricator to RW
  • check everything is ok
  • close this task

Mentioned in SAL (#wikimedia-operations) [2021-01-27T06:00:54Z] <marostegui> m3 master restart, phabricator will go on read only - T272596

This was done. RO time for phabricator was around:

06:01:53 ON
06:03:12 OFF

Thanks @mmodell for the help!

root@db1132.eqiad.wmnet[(none)]> select @@report_host;
+--------------------+
| @@report_host      |
+--------------------+
| db1132.eqiad.wmnet |
+--------------------+
1 row in set (0.001 sec)

Mentioned in SAL (#wikimedia-operations) [2021-01-28T09:00:11Z] <marostegui> m5 master restart, wikitech will be unavailable - T272596