Page MenuHomePhabricator

Lumen eqiad<->codfw transport outage (Aug 2026).
Closed, ResolvedPublic

Assigned To
Authored By
phaultfinder
Aug 24 2026, 12:28 PM
Referenced Files
Restricted File
Aug 28 2026, 4:42 AM
F100417236: lumen_cct_improvement_2026.08.28.png
Aug 28 2026, 4:38 AM
F100417129: lumen_cct_light_level_bounce_ashburn_2026-08-28.png
Aug 28 2026, 4:38 AM
F99849806: image.png
Aug 24 2026, 1:53 PM
F99847131: image.png
Aug 24 2026, 1:25 PM
F99846500: image.png
Aug 24 2026, 1:25 PM

Description

Below automated task was created on the back of errors detected on the link. Further examination (see first comment) reveals the circuit is bouncing up and down frequently, optic reports -40dBm light regularly.

The link has been drained and we are following up with the carrier.

Common information

  • alertname: OutboundInterfaceErrors
  • instance: cr2-eqiad:9804
  • interface_description: Transport: cr2-codfw:et-0/1/4 (Lumen, 449169461)
  • interface_name: et-1/1/5
  • job: gnmi
  • prometheus: ops
  • scope: network
  • severity: task
  • site: eqiad
  • source: prometheus
  • team: dcops

Firing alerts


Event Timeline

Mentioned in SAL (#wikimedia-operations) [2026-08-24T13:12:32Z] <topranks> drain Lumen transport circuit from eqiad<->codfw due to link flaps/errors (T435810)

This link has been flapping up and down repeatedly, light coming/going on the interface

image.png (921×433 px, 59 KB)

First sign of problems seem to have been just after midnight UTC, but have gotten significantly worse since ~12:00 today:

image.png (1,774×863 px, 118 KB)

I can't see any maintenance in the calendar or email for this one, I will raise a ticket with Lumen about the issue. For now the link has been drained and so it should not cause any further operational problems.

cmooney renamed this task from Outbound errors on interface cr2-eqiad:et-1/1/5 (Transport: cr2-codfw:et-0/1/4 (Lumen, 449169461)) to Lumen eqiad<->codfw transport instability (Aug 2026)..Aug 24 2026, 1:27 PM
cmooney updated the task description. (Show Details)

Ticket ID 35191415 opened with Lumen.

FWIW this led to a page for dropped ping to db2236, and we can also see the effects in the blackbox probes graphs:

image.png (1,774×863 px, 179 KB)

Lumen came back to say they found a problem and are working on it.

On Mon, 24 Aug 2026 at 19:50, <ticketing_noreply@lumen.com> wrote:

Trouble shooting the circuit path, the issue appears to be in the indianapolis gateway, there is a loss of frame that cannot clear with a loop. A dispatch has been requested. No ETA
cmooney renamed this task from Lumen eqiad<->codfw transport instability (Aug 2026). to Lumen eqiad<->codfw transport ~~instability~~ outage (Aug 2026)..Aug 24 2026, 7:07 PM
cmooney renamed this task from Lumen eqiad<->codfw transport ~~instability~~ outage (Aug 2026). to Lumen eqiad<->codfw transport outage (Aug 2026)..
On Mon, 24 Aug 2026 at 20:18, <ticketing_noreply@lumen.com> wrote:

    The Lumen field tech is now on site and we are working with them remotely, more status will be provided shortly.
The Lumen NOC has reported that an equipment issue will be corrected during an Urgent Maintenance Network Event on August 25, 2026, between 05:00 GMT and 11:00 GMT. The Lumen NOC detected errors causing unprotected customer impairment and leaving services at risk of failure. As such, this activity will be implemented to reset the affected device and prevent future service interruptions. This is a preliminary notification and the next update will be sent at the commencement of the maintenance.
2026-08-25 06:46:48 GMT - Due to unforeseen circumstances, the Lumen NOC has advised that this Urgent Maintenance Network Event could not be completed during the scheduled maintenance window. As such, the necessary repair will be scheduled under a planned maintenance on a future, yet to be determined date. Services that are expected to be impacted will receive notifications advising when the permanent repair will be performed.

2026-08-25 05:19:30 GMT - The Lumen NOC has confirmed the Urgent Maintenance Network Event to address a card issue in Kansas City, MO is commencing at this time. Please be advised the next update will be provided once the Lumen NOC the Urgent Maintenance Network Event has been completed, or if new information becomes available.

I updated the ticket to say we are effectively hard down and need them to treat this as an emergency fix situation, not some leisurely "planned maintenance".

They came back with the same update again this evening. Sigh.

Hi Tim,

With all due respect that is the update from 11 hours ago!

Our service has been down for over 36 hours at this point, and Lumen's response is "we will fix it at some point in the future, but not now"?!?

That really is unacceptable from a customer point of view.  We need the service we are paying for to be operational.  At very least I would expect detail of what the plan is to get it to that state.

thanks,

Cathal.

They seemed to have pulled their finger out now.

2026-08-25 20:22:05 GMT - The maintenance window starts at 12am CST. We will follow up again with the TNOC once the maintenance window starts.
2026-08-26 05:08:20 GMT - The Lumen NOC has confirmed the Urgent Maintenance Network Event to address a card issue in Kansas City, MO is commencing at this time. Please be advised the next update will be provided once the Lumen NOC the Urgent Maintenance Network Event has been completed, or if new information becomes available.

2026-08-26 05:07:45 GMT - The Lumen NOC has reported that an equipment issue will be corrected during an Urgent Maintenance Network Event on August 26, 2026, between 05:00 GMT and 11:00 GMT. The Lumen NOC detected errors causing unprotected customer impairment and leaving services at risk of failure. As such, this activity will be implemented to reset the affected device and prevent future service interruptions. This is a preliminary notification and the next update will be sent at the commencement of the maintenance.
Hi,

Please note the issue has not resolved for us.  The interface has flapped down over 2,000 times since 05:56 UTC this morning and remains very unstable, though it has improved slightly since ~07:45 UTC.  Please see attached file lumen_link_flaps_26.08.2026.png

Can you provide an update when available?

thanks,

Cathal.


*** CASCADED EXTERNAL NOTES 2026-08-26 08:39:48 GMT Ramirez From CASE: 35192092 - Event
Event Conclusion Summary: 

Outage Start: August 26, 2026, 05:12 GMT
Outage Stop: August 26, 2026, 05:56 GMT

Root Cause: An Urgent Maintenance Network event to address a card issue in Kansas City, MO impacted client services.

Fix Action: The Urgent Maintenance Network Event was completed, thus restoring services to a stable state.

Reason for Outage (RFO) Summary: The Lumen NOC reported a card issue in Kansas City, MO would be addressed during an Urgent Maintenance Network Event on  August 26, 2026 between 05:00 GMT and 11:00 GMT. The Lumen NOC detected errors causing unprotected customer impairment and leaving services at risk. As such, this activity was necessary to prevent an unplanned service interruption in the future. On August 26, 2026, at 05:12 GMT the Lumen NOC confirmed the Urgent Maintenance Network Event had commenced. The card replacement was completed on August 26, 2026, at 05:56 GMT, thus restoring services to a stable state.

This service impact has concluded. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment. If after that action has been performed a service issue prevails, please contact the Lumen Repair Center for troubleshooting assistance.

2026-08-26 05:08:20 GMT - The Lumen NOC has confirmed the Urgent Maintenance Network Event to address a card issue in Kansas City, MO is commencing at this time. Please be advised the next update will be provided once the Lumen NOC the Urgent Maintenance Network Event has been completed, or if new information becomes available.

Lumen have come back twice to say everything is ok still. I have in both cases responded to them showing logs and letting them know we need the service to work and it's clearly not.

Lumen did not respond to my updates on the ticket and closed it as "resolved".

I opened another ticket with them ref #35211090.

@VRiley-WMF I've been having it out with Lumen on this one. I still think the issue is their side, but if possible can you try to see if we have a spare 100GBase-LR4 optic (blue handle) that we can maybe swap in port et-1/1/5 on cr2-eqiad?

The port/circuit is not being used right now, so it can be swapped any time. You should see the light level on the below graph after the swap to confirm things are ok:

https://grafana.wikimedia.org/d/5p97dAASz/queue-and-error-stats-by-network-device?from=now-6h&to=now&timezone=utc&var-site=000000006&var-device=cr2-eqiad:9804&var-interface=et-1%2F1%2F5&refresh=30s&theme=dark

I'm unfortunately out now for the next few days but it would be best to get this done if we can. Thanks.

Okay, no problem. I'll take a look at it now.

I just swapped out the optic. Looking at grafana now...

@VRiley-WMF thanks, light levels are ok, doesn't seem to be any change in the link flapping but I wasn't expecting it really. Please put the optic that came out in spares, I suspect it is fine.

Lumen did some work overnight it seems:

2026-08-28 02:33:43 GMT - Lumen observed a higher-level issue in Columbus, OH on 8/24 that bounced 15times but has been clear since. We also observed a higher-level issue in KANSAS CITY, MO on 8/25. I did observe 3 errors today on a port in INDIANAPOLIS, IN I rebuilt the transceiver on one and the port on the box facing it. We will continue to monitor the path.

While that doesn't sound too promising we seem to have had a recovery. The light level dropped off completely for about an hour earlier, prior to the above update:

lumen_cct_light_level_bounce_ashburn_2026-08-28.png (1,831×883 px, 122 KB)

When it came back up the link flapping ceased:

lumen_cct_improvement_2026.08.28.png (1,831×883 px, 109 KB)

Link shows no loss since:

{F100417672 width=700}

So whatever they did it seems to have fixed it. I'll update the ticket to say as much.

As the recovery is recent and it's been problematic I won't repool today (it's also Friday and neither myself or Arzhel are working). @ayounsi can you undrain Monday if it remains clean? Thanks.

OSPF was set to 5000 only on one side, I normalized it back to 500 now that the link has been stable.

Jclark-ctr claimed this task.
Jclark-ctr subscribed.

Confirmed with ayounsi looks healthy. Resolving for now will reopen if it returns

Mentioned in SAL (#wikimedia-operations) [2026-09-03T09:15:37Z] <topranks> put traffic on Lumen codfw<->eqiad link as it is stable T435810