Page MenuHomePhabricator

Upgrade mysql on db1128 (m5 db master)
Closed, ResolvedPublic

Description

We need to upgrade mysql on db1128 (m5 master) due to an upstream bug see T279281
This is simply a mysql restart, so I expect it to take around 1 minute
Impact: Neither writes or reads will work for that period, as m5 doesn't use a proxy.

Affected databases:

labsdbaccounts
labswiki
striker
test_labsdbaccounts
testmailman3
testmailman3web

@Andrew @Ladsgroup @Legoktm I would like to this sometime next week.
@Andrew this means downtime for wikitech, but also for the other databases, would you tell me which day/time would work best for you?
@Ladsgroup @Legoktm can you make sure no imports or other testing would be on-going once we've decided on a day/time?

Thanks

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui moved this task from Triage to Refine on the DBA board.

@Ladsgroup @Legoktm can you make sure no imports or other testing would be on-going once we've decided on a day/time?

Yep!

That's ok for us!

Ping me on IRC and I will:

  • downtime labstore1004
  • stop puppet
  • shutdown maintain-dbusers

Thanks @arturo - what about Monday 19th at 09:00 AM UTC?

Thanks @arturo - what about Monday 19th at 09:00 AM UTC?

works for me, thanks! please ping me or @dcaro on IRC to make sure we are in front of the laptop :-)

Icinga downtime set by dcaro@cumin1001 for 2:00:00 1 host(s) and their services with reason: Restarting mysql

labstore1004.eqiad.wmnet

Mentioned in SAL (#wikimedia-cloud) [2021-04-19T08:09:27Z] <dcaro> downtiming labstore1004 and stopping puppet for mysql restart (T279657)

Mentioned in SAL (#wikimedia-operations) [2021-04-19T08:30:56Z] <marostegui> Restart m5 master - wikitech will go down T279657

This has been done.
Mysql was down from 08:31:00 to 08:31:32

Thanks @dcaro and @aborrero for the support!

Mentioned in SAL (#wikimedia-cloud) [2021-04-19T08:40:33Z] <dcaro> enabling puppet on labstore1004 after mysql restart (T279657)