Page MenuHomePhabricator

Reclone labsdb1011
Closed, ResolvedPublic

Description

Due to the recent crashes of labsdb1011 discussed at T233986, this host has shown signs of data corruption and needs to be recloned.
It will be recloned from labsdb1012 once Analytics gives us greenlight to stop MySQL there.

Event Timeline

Marostegui moved this task from Triage to Pending comment on the DBA board.

If everything goes as planned, we are expecting to be able to start the recloning around the 10th Oct

Change 542000 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1011: Depool labsdb1011 to reclone it from labsdb1012

https://gerrit.wikimedia.org/r/542000

Change 542000 merged by Marostegui:
[operations/puppet@production] dbproxy1011: Depool labsdb1011 to reclone it from labsdb1012

https://gerrit.wikimedia.org/r/542000

Mentioned in SAL (#wikimedia-operations) [2019-10-10T04:43:58Z] <marostegui> Depool labsdb1011 for recloning - T235016

labsdb1011 has been recloned. I am letting it to catch up a bit (it is 7h delayed) before repooling it.

Mentioned in SAL (#wikimedia-operations) [2019-10-10T13:27:25Z] <marostegui> Repool labsdb1011 after reclone - T235016

Marostegui mentioned this in Unknown Object (Task).Oct 21 2019, 7:52 AM