The eqiad - eqord Telia link went down and it did not look planned. There was no mail from Telia regarding IC-314533 since Jan 9.
Current Status: CRITICAL
Status Information: CRITICAL: host '208.80.154.197', interfaces up: 229, down: 1, dormant: 0, excluded: 0, unused: 0:
xe-4/2/0: down -> Transport: cr2-eqord:xe-0/1/5 (Telia, IC-314533, 29ms) {#3658} [10Gbps wave];
https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down
Last State Change: 2019-03-14 14:55:31
I sent an email to Telia with the circuit ID and time. They responded saying "Its up now ? There is no scheduled maintenance from our side. " and i said No, it's still down, our monitoring shows it down on both ends.
Telia claims the issue was resolved. our monitoring can't confirm that.
Telia Carrier Case Reference 00959377 Service ID IC-314533 Service Type DWDM - 10 Gigabit - Ashburn - Chicago Case Opened 2019-03-14 15:38 UTC Case Resolved 2019-03-14 15:38 UTC Summary We checked and there was a LOS from your side in Ashburn. Please let us know if you need any further assistance from us.
We're getting one way light, followed up with Telia.
Ashburn side:
Physical interface: xe-4/2/0
Laser bias current : 43.070 mA
Laser output power : 0.7540 mW / -1.23 dBm
Receiver signal average optical power : 0.3297 mW / -4.82 dBmChicago side:
Physical interface: xe-0/1/5
Laser bias current : 45.520 mA
Laser output power : 0.7260 mW / -1.39 dBm
Laser receiver power : 0.0001 mW / -40.00 dBmTelia did a loop test facing eqiad and our light levels didn't change. While Telia still don't receive light.
The culprit seems to be an active element somewhere on the cr2-eqiad (in DC6)<->Telia (in DC2) X-connect path.
Link is back up. Cf. email thread for full details. But tl;dr; outdated cable label caused onsite tech to unplug our link.