Down are:
- 2 of the 3 links between cr1 and cr2,
- one of the transports to codfw,
- one of the telia transit links
- one of the Equinix peering links
Everything failed over as expected, opening a JTAC case.
cr2-eqiad> show system alarms 1 alarms currently active Alarm time Class Description 2022-07-10 23:43:46 UTC Major FPC 3 Major Errors - XM Chip Error code: 0x70139
Jul 10 23:43:30 re0.cr2-eqiad fpc3 XMCHIP(1): XMCHIP(1): FI: Reorder cell timeout - Stream 12, Count 6 Jul 10 23:43:30 re0.cr2-eqiad fpc3 XMCHIP(1): XMCHIP(1): FI: Packet CRC error - Stream 12, Count 1 Jul 10 23:43:46 re0.cr2-eqiad fpc3 XMCHIP(1): XMCHIP(1): FI: Reorder cell timeout - Stream 12, Count 5 Jul 10 23:43:46 re0.cr2-eqiad fpc3 XMCHIP(1): XMCHIP(1): FI: Packet CRC error - Stream 19, Count 2 Jul 10 23:43:46 re0.cr2-eqiad fpc3 XMCHIP(1): XMCHIP(1): FI: Aliasing on allocates error - Pipe count 0, Count 4 Jul 10 23:43:46 re0.cr2-eqiad fpc3 XMCHIP(1): XMCHIP(1): FI: Cells dropped due to reorder sequence jumping - Count 1 Jul 10 23:43:47 re0.cr2-eqiad fpc3 Cmerror Op Set: XMCHIP(1): XMCHIP(1): FI: Link sanity checks - Type 4, Seq Number 646, Stream 150, Link0 0x4, Link1 0x10, Link2 0xfff Jul 10 23:43:47 re0.cr2-eqiad fpc3 Error (0x70139), module: XMCHIP(1), type: FI: Link sanity check and high rate cell underflow errors Jul 10 23:43:48 re0.cr2-eqiad fpc3 XMCHIP(1): XMCHIP(1): FI: Aliasing on allocates error - Pipe count 0, Count 1 Jul 10 23:43:48 re0.cr2-eqiad fpc3 XMCHIP(1): XMCHIP(1): FI: Aliasing on allocates error - Pipe count 0, Count 1 Jul 10 23:44:00 re0.cr2-eqiad fpc3 PFE 1: 'PFE Disable' action performed. Bringing down ifd xe-3/2/0 191 Jul 10 23:44:00 re0.cr2-eqiad fpc3 PFE 1: 'PFE Disable' action performed. Bringing down ifd xe-3/2/1 192 Jul 10 23:44:00 re0.cr2-eqiad fpc3 PFE 1: 'PFE Disable' action performed. Bringing down ifd xe-3/2/2 193 [...]