Due to the recent crashes of labsdb1011 discussed at T233986, this host has shown signs of data corruption and needs to be recloned.
It will be recloned from labsdb1012 once Analytics gives us greenlight to stop MySQL there.
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| dbproxy1011: Depool labsdb1011 to reclone it from labsdb1012 | operations/puppet | production | +6 -6 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Marostegui | T233766 labsdb1011 mariadb crashed | |||
| Restricted Task | |||||
| Resolved | • Marostegui | T235016 Reclone labsdb1011 |
Event Timeline
Comment Actions
If everything goes as planned, we are expecting to be able to start the recloning around the 10th Oct
Comment Actions
Change 542000 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1011: Depool labsdb1011 to reclone it from labsdb1012
Comment Actions
Change 542000 merged by Marostegui:
[operations/puppet@production] dbproxy1011: Depool labsdb1011 to reclone it from labsdb1012
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-10-10T04:43:58Z] <marostegui> Depool labsdb1011 for recloning - T235016
Comment Actions
labsdb1011 has been recloned. I am letting it to catch up a bit (it is 7h delayed) before repooling it.
Comment Actions
For the record I have also documented briefily how to reclone one of the wikireplicas: https://wikitech.wikimedia.org/w/index.php?title=MariaDB&diff=prev&oldid=1840626#Recloning_a_Wiki_replica
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-10-10T13:27:25Z] <marostegui> Repool labsdb1011 after reclone - T235016