The core routers in drmrs were recently upgraded to 23.4R2-S7.4. Unfortunately this version is vulnerable to the bug that crashes rpd when the graceful-shtudown sender command is used, so we will need to upgrade them again to 23.4R2-S8.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Open | None | T416444 2026 Junos upgrade | |||
| Resolved | cmooney | T429386 cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) | |||
| Open | None | T431748 Core router upgrades 2026 #2 | |||
| Resolved | • ayounsi | T431749 drmrs: upgrade core routers (2026 #2) |
Mentioned in SAL (#wikimedia-operations) [2026-07-28T08:05:34Z] <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: router upgrade, T431749]
Mentioned in SAL (#wikimedia-operations) [2026-07-28T08:05:39Z] <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: router upgrade, T431749]
Icinga downtime and Alertmanager silence (ID=8089257c-9548-4d05-a03b-bdb3c1224c47) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their services with reason: router upgrade
cr1-drmrs,cr1-drmrs IPv6,cr1-drmrs.mgmt
Mentioned in SAL (#wikimedia-operations) [2026-07-28T08:33:14Z] <XioNoX> draining cr1-drmrs - T431749
Mentioned in SAL (#wikimedia-operations) [2026-07-28T08:35:54Z] <XioNoX> rebooting cr1-drmrs - T431749
Mentioned in SAL (#wikimedia-operations) [2026-07-28T08:48:09Z] <XioNoX> un-drain cr1-drmrs - T431749
Mentioned in SAL (#wikimedia-operations) [2026-07-28T09:06:56Z] <XioNoX> draining cr2-drmrs - T431749
Icinga downtime and Alertmanager silence (ID=5c71b24d-45b0-4c80-bf42-231dd7bab42a) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their services with reason: router upgrade
cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt
Mentioned in SAL (#wikimedia-operations) [2026-07-28T09:11:04Z] <XioNoX> rebooting cr2-drmrs - T431749
Mentioned in SAL (#wikimedia-operations) [2026-07-28T09:30:36Z] <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: router upgrade, T431749]
Mentioned in SAL (#wikimedia-operations) [2026-07-28T09:30:41Z] <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: router upgrade, T431749]
Mentioned in SAL (#wikimedia-operations) [2026-07-28T13:28:57Z] <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: depool esams [reason: router upgrade, T431749]
Mentioned in SAL (#wikimedia-operations) [2026-07-28T13:29:03Z] <ayounsi@cumin1003> END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool esams [reason: router upgrade, T431749]