Status	Assigned	Task
		Restricted Task
Stalled	None	T439136 Juniper hostkey-algorithm deprecation
Open	None	T416444 2026 Junos upgrade
Open	colewhite	T390215 Logstash is still overwhelmed (since March 2025)
Open	ayounsi	T412143 ~5k/logs/sec from netdev
Resolved	ayounsi	T416450 esams: upgrade routers & switches (2026)
Resolved	ayounsi	T422525 cr1-esams failed upgrade

Event Timeline

ayounsi created this task.Feb 4 2026, 9:02 AM

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptFeb 4 2026, 9:02 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

ayounsi added a parent task: T416444: 2026 Junos upgrade.Feb 4 2026, 9:40 AM

ayounsi added a parent task: T412143: ~5k/logs/sec from netdev.

ayounsi triaged this task as Medium priority.Feb 9 2026, 3:39 PM

Scheduling this for April 7th at 12:00 UTC - 2h
Pinging @ssingh (Traffic) for visibility.

And doing mr1-esams now to get it out of the way.

Icinga downtime and Alertmanager silence (ID=7822bc9e-76f4-4f51-943c-ae5d8f2f7739) set by ayounsi@cumin1003 for 0:30:00 on 4 host(s) and their services with reason: mgmt router upgrade

mr1-esams,mr1-esams IPv6,mr1-esams.oob,mr1-esams.oob IPv6

Mentioned in SAL (#wikimedia-operations) [2026-04-02T08:42:00Z] <XioNoX> reboot mr1-esams - T416450

ayounsi updated the task description. (Show Details)Apr 2 2026, 8:52 AM

In T416450#11781010, @ayounsi wrote:

Scheduling this for April 7th at 12:00 UTC - 2h
Pinging @ssingh (Traffic) for visibility.

And doing mr1-esams now to get it out of the way.

Sounds good, thanks for sending the email out as well!

Mentioned in SAL (#wikimedia-operations) [2026-04-07T11:36:10Z] <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: depool esams [reason: network maintenance, T416450]

Mentioned in SAL (#wikimedia-operations) [2026-04-07T11:36:13Z] <XioNoX> depool esams for network maintenance - T416450

Mentioned in SAL (#wikimedia-operations) [2026-04-07T11:36:18Z] <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: network maintenance, T416450]

Change #1268538 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/dns@master] Temporarily geodns GB and IE to eqiad

https://gerrit.wikimedia.org/r/1268538

gerritbot added a project: Patch-For-Review.Apr 7 2026, 11:39 AM

Icinga downtime and Alertmanager silence (ID=5c0f0433-7743-49c9-b411-8f120a9f337d) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their services with reason: router upgrade

cr1-esams,cr1-esams IPv6,re0.cr1-esams.mgmt

Mentioned in SAL (#wikimedia-operations) [2026-04-07T12:04:47Z] <XioNoX> reboot re1.cr1-esams (backup RE) for upgrade - T416450

Mentioned in SAL (#wikimedia-operations) [2026-04-07T12:11:45Z] <XioNoX> re0.cr1-esams> request chassis routing-engine master switch - that will cause router's short unavailability - T416450

Mentioned in SAL (#wikimedia-operations) [2026-04-07T12:39:41Z] <XioNoX> re1.cr1-esams> request chassis routing-engine master switch - that will cause router's short unavailability - T416450

Icinga downtime and Alertmanager silence (ID=3c9a80b3-71a3-420a-bae4-d8cf79e5188e) set by ayounsi@cumin1003 for 0:30:00 on 3 host(s) and their services with reason: router upgrade

cr1-esams,cr1-esams IPv6,re0.cr1-esams.mgmt

Icinga downtime and Alertmanager silence (ID=b81e6903-f6db-4d47-bfbb-a5bff02b24fa) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their services with reason: router upgrade

cr1-esams,cr1-esams IPv6,re0.cr1-esams.mgmt

Icinga downtime and Alertmanager silence (ID=918af1bd-6f87-4ec9-acb7-39622f38db7c) set by ayounsi@cumin1003 for 0:30:00 on 3 host(s) and their services with reason: router upgrade

cr1-esams,cr1-esams IPv6,re0.cr1-esams.mgmt

Mentioned in SAL (#wikimedia-operations) [2026-04-07T15:28:57Z] <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: pool esams [reason: network maintenance over, T416450]

Mentioned in SAL (#wikimedia-operations) [2026-04-07T15:29:00Z] <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: network maintenance over, T416450]

cmooney added a subtask: T422525: cr1-esams failed upgrade.Apr 7 2026, 3:55 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-14T11:45:59Z] <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: depool esams [reason: network maintenance, T416450]

Mentioned in SAL (#wikimedia-operations) [2026-04-14T11:46:33Z] <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: network maintenance, T416450]

Icinga downtime and Alertmanager silence (ID=eea64f46-a692-4b84-9bd8-3c707fabc91e) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their services with reason: router upgrade

cr1-esams,cr1-esams IPv6,re0.cr1-esams.mgmt

Mentioned in SAL (#wikimedia-operations) [2026-04-14T12:17:52Z] <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: pool esams [reason: network maintenance paused, T416450]

Mentioned in SAL (#wikimedia-operations) [2026-04-14T12:17:55Z] <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: network maintenance paused, T416450]

Mentioned in SAL (#wikimedia-operations) [2026-04-14T12:22:12Z] <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: depool esams [reason: network maintenance continue, T416450]

Mentioned in SAL (#wikimedia-operations) [2026-04-14T12:22:15Z] <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: network maintenance continue, T416450]

Mentioned in SAL (#wikimedia-operations) [2026-04-14T12:33:52Z] <XioNoX> cr1-esams - request chassis routing-engine master switch - T416450

Icinga downtime and Alertmanager silence (ID=969c7620-d5fd-476a-bc01-620cf3273f2e) set by ayounsi@cumin1003 for 0:30:00 on 3 host(s) and their services with reason: router upgrade

cr1-esams,cr1-esams IPv6,re0.cr1-esams.mgmt

Mentioned in SAL (#wikimedia-operations) [2026-04-14T13:01:06Z] <XioNoX> cr1-esams - request chassis routing-engine master switch - T416450

Icinga downtime and Alertmanager silence (ID=8f6b7eae-ec0e-422c-9012-4219cdb0af72) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their services with reason: router upgrade

cr2-esams,cr2-esams IPv6,cr2-esams.mgmt

Mentioned in SAL (#wikimedia-operations) [2026-04-14T13:15:34Z] <XioNoX> cr2-esams - request vmhost reboot - T416450

Icinga downtime and Alertmanager silence (ID=318be034-a1ac-42f8-a346-f076fb030f09) set by ayounsi@cumin1003 for 1:00:00 on 13 host(s) and their services with reason: router upgrade

cp[3066,3068,3070,3072,3074,3076,3078,3080].esams.wmnet,dns3004.wikimedia.org,ganeti[3006,3008].esams.wmnet,lvs[3008,3010].esams.wmnet

Icinga downtime and Alertmanager silence (ID=7aa6ddc4-76ea-4040-9c97-e3ee3ff2982d) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their services with reason: router upgrade

asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt

Mentioned in SAL (#wikimedia-operations) [2026-04-14T13:36:07Z] <XioNoX> asw1-bw27-esams> request system reboot - T416450

Icinga downtime and Alertmanager silence (ID=60febde6-0fbd-4034-aa7f-25aa1b23e19d) set by ayounsi@cumin1003 for 1:00:00 on 12 host(s) and their services with reason: router upgrade

cp[3067,3069,3071,3073,3075,3077,3079,3081].esams.wmnet,dns3003.wikimedia.org,ganeti[3005,3007].esams.wmnet,lvs3009.esams.wmnet

Icinga downtime and Alertmanager silence (ID=d98c0310-2bee-4b0c-9c4f-43b3ad5852ad) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their services with reason: router upgrade

asw1-by27-esams,asw1-by27-esams IPv6,asw1-by27-esams.mgmt

Icinga downtime and Alertmanager silence (ID=3ae4af28-cae2-4928-bd05-37949f6ccde3) set by ayounsi@cumin1003 for 1:00:00 on 8 host(s) and their services with reason: router upgrade

bast3007.wikimedia.org,doh3005.wikimedia.org,hcaptcha-proxy[3001-3002].wikimedia.org,install3004.wikimedia.org,ncredir3006.esams.wmnet,netflow3004.esams.wmnet,prometheus3004.esams.wmnet

Mentioned in SAL (#wikimedia-operations) [2026-04-14T13:57:26Z] <XioNoX> asw1-by27-esams> request system reboot - T416450

Mentioned in SAL (#wikimedia-operations) [2026-04-14T14:16:21Z] <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: pool esams [reason: network maintenance finished, T416450]

Mentioned in SAL (#wikimedia-operations) [2026-04-14T14:16:25Z] <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: network maintenance finished, T416450]