The host crashed due to T434532: reimage it, put it back into production and reprovision its data for extra redundancy of database backups. CC @Marostegui just for awareness, nothing required from you atm.
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Jhancock.wm | T434532 db2201 memory failure (was: both mysql instances crashed in the last few days) | |||
| Resolved | Marostegui | T437411 Put db2201 back into backup production as a backup source |
Event Timeline
Comment Actions
Change to be reverted whenever ready: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324672
Comment Actions
Change #1342596 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2201: Enable notifications
Comment Actions
Change #1342596 merged by Marostegui:
[operations/puppet@production] db2201: Enable notifications
Comment Actions
Change #1342598 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2250: Enable notifications
Comment Actions
Change #1342598 merged by Marostegui:
[operations/puppet@production] db2250: Enable notifications
Comment Actions
db2201 has been added back in the backups rotation for s5.
Once confirmed backups work fine, I have to remove s5 from db2250 to leave that config on how it originally was before this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324577
Comment Actions
This has worked fine - I will work for the s5 dump to be also done correctly before proceeding with the removal in db2250 (dumps happen on Tuesdays):
| 42576 | snapshot.s5.2026-09-18--20-00-01 | finished | db2201.codfw.wmnet:3315 | dbprov2005.codfw.wmnet | snapshot | s5 | 2026-09-18 20:37:15 | 2026-09-18 21:14:16 | 579211839708 | | 42596 | snapshot.s5.2026-09-20--20-00-01 | finished | db2201.codfw.wmnet:3315 | dbprov2005.codfw.wmnet | snapshot | s5 | 2026-09-20 20:36:08 | 2026-09-20 21:14:38 | 579399289055 | +-------+----------------------------------+----------+-------------------------+------------------------+----------+---------+---------------------+---------------------+--------------+
Comment Actions
| 42613 | dump.s5.2026-09-22--00-00-02 | finished | db2201.codfw.wmnet:3315 | dbprov2005.codfw.wmnet | dump | s5 | 2026-09-22 00:00:02 | 2026-09-22 00:32:22 | 56414782294 |
The logical backups also worked fine.
Comment Actions
Change #1343932 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2250: Remove s5
Comment Actions
Change #1343932 merged by Marostegui:
[operations/puppet@production] db2250: Remove s5
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2026-09-22T12:44:00Z] <marostegui> Stop mariadb on db2250:s5 T437411 T437279
Comment Actions
Mariadb has been stopped on db2250:s5
I am going to consider this resolved and the follow ups are on T437279