Page MenuHomePhabricator

Reclone labsdb1011
Closed, ResolvedPublic

Description

Due to the recent crashes of labsdb1011 discussed at T233986, this host has shown signs of data corruption and needs to be recloned.
It will be recloned from labsdb1012 once Analytics gives us greenlight to stop MySQL there.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Oct 9, 8:39 AM
Marostegui triaged this task as High priority.Wed, Oct 9, 8:39 AM
Marostegui moved this task from Triage to Next on the DBA board.

If everything goes as planned, we are expecting to be able to start the recloning around the 10th Oct

Change 542000 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1011: Depool labsdb1011 to reclone it from labsdb1012

https://gerrit.wikimedia.org/r/542000

Change 542000 merged by Marostegui:
[operations/puppet@production] dbproxy1011: Depool labsdb1011 to reclone it from labsdb1012

https://gerrit.wikimedia.org/r/542000

Mentioned in SAL (#wikimedia-operations) [2019-10-10T04:43:58Z] <marostegui> Depool labsdb1011 for recloning - T235016

Marostegui moved this task from Next to In progress on the DBA board.Thu, Oct 10, 5:07 AM

labsdb1011 has been recloned. I am letting it to catch up a bit (it is 7h delayed) before repooling it.

For the record I have also documented briefily how to reclone one of the wikireplicas: https://wikitech.wikimedia.org/w/index.php?title=MariaDB&diff=prev&oldid=1840626#Recloning_a_Wiki_replica

Mentioned in SAL (#wikimedia-operations) [2019-10-10T13:27:25Z] <marostegui> Repool labsdb1011 after reclone - T235016

Marostegui closed this task as Resolved.Thu, Oct 10, 1:55 PM

Host repooled