At 13:08 UTC today Icinga reported:
cr1-eqiad: CRITICAL: host '208.80.154.196', interfaces up: 239, down: 1, dormant: 0, excluded: 0, unused: 0: xe-4/2/0: down -> Transport: cr1-codfw:xe-5/2/1 (Telia, IC-307235, 34ms) {#2648} [10Gbps wave];
and at the same time cr1-codfw reported a critical OSPF status, but does not report interface down. So this probably suggests the issue with the link is on the eqiad side?
Description
Description
Event Timeline
Comment Actions
volans@re0.cr1-eqiad> show interfaces diagnostics optics xe-4/2/0
Physical interface: xe-4/2/0
Laser bias current : 39.156 mA
Laser output power : 0.7330 mW / -1.35 dBm
Module temperature : 40 degrees C / 104 degrees F
Module voltage : 3.3000 V
Receiver signal average optical power : 0.0002 mW / -36.99 dBm
Laser bias current high alarm : Off
Laser bias current low alarm : Off
Laser bias current high warning : Off
Laser bias current low warning : Off
Laser output power high alarm : Off
Laser output power low alarm : Off
Laser output power high warning : Off
Laser output power low warning : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser rx power high alarm : Off
Laser rx power low alarm : On
Laser rx power high warning : Off
Laser rx power low warning : On
Laser bias current high alarm threshold : 85.000 mA
Laser bias current low alarm threshold : 15.000 mA
Laser bias current high warning threshold : 80.000 mA
Laser bias current low warning threshold : 20.000 mA
Laser output power high alarm threshold : 1.5840 mW / 2.00 dBm
Laser output power low alarm threshold : 0.1580 mW / -8.01 dBm
Laser output power high warning threshold : 1.2580 mW / 1.00 dBm
Laser output power low warning threshold : 0.1990 mW / -7.01 dBm
Module temperature high alarm threshold : 78 degrees C / 172 degrees F
Module temperature low alarm threshold : -13 degrees C / 9 degrees F
Module temperature high warning threshold : 73 degrees C / 163 degrees F
Module temperature low warning threshold : -8 degrees C / 18 degrees F
Module voltage high alarm threshold : 3.700 V
Module voltage low alarm threshold : 2.900 V
Module voltage high warning threshold : 3.600 V
Module voltage low warning threshold : 3.000 V
Laser rx power high alarm threshold : 1.7783 mW / 2.50 dBm
Laser rx power low alarm threshold : 0.0100 mW / -20.00 dBm
Laser rx power high warning threshold : 1.5849 mW / 2.00 dBm
Laser rx power low warning threshold : 0.0158 mW / -18.01 dBmComment Actions
volans@re0.cr1-codfw> show interfaces diagnostics optics xe-5/2/1
Physical interface: xe-5/2/1
Laser bias current : 40.898 mA
Laser output power : 0.7040 mW / -1.52 dBm
Module temperature : 31 degrees C / 88 degrees F
Module voltage : 3.2810 V
Receiver signal average optical power : 0.3757 mW / -4.25 dBm
Laser bias current high alarm : Off
Laser bias current low alarm : Off
Laser bias current high warning : Off
Laser bias current low warning : Off
Laser output power high alarm : Off
Laser output power low alarm : Off
Laser output power high warning : Off
Laser output power low warning : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser rx power high alarm : Off
Laser rx power low alarm : Off
Laser rx power high warning : Off
Laser rx power low warning : Off
Laser bias current high alarm threshold : 85.000 mA
Laser bias current low alarm threshold : 15.000 mA
Laser bias current high warning threshold : 80.000 mA
Laser bias current low warning threshold : 20.000 mA
Laser output power high alarm threshold : 1.5840 mW / 2.00 dBm
Laser output power low alarm threshold : 0.1580 mW / -8.01 dBm
Laser output power high warning threshold : 1.2580 mW / 1.00 dBm
Laser output power low warning threshold : 0.1990 mW / -7.01 dBm
Module temperature high alarm threshold : 78 degrees C / 172 degrees F
Module temperature low alarm threshold : -13 degrees C / 9 degrees F
Module temperature high warning threshold : 73 degrees C / 163 degrees F
Module temperature low warning threshold : -8 degrees C / 18 degrees F
Module voltage high alarm threshold : 3.700 V
Module voltage low alarm threshold : 2.900 V
Module voltage high warning threshold : 3.600 V
Module voltage low warning threshold : 3.000 V
Laser rx power high alarm threshold : 1.7783 mW / 2.50 dBm
Laser rx power low alarm threshold : 0.0100 mW / -20.00 dBm
Laser rx power high warning threshold : 1.5849 mW / 2.00 dBm
Laser rx power low warning threshold : 0.0158 mW / -18.01 dBmComment Actions
Telia reports a 'major outage' and is tracking status of our circuit in case 00993514
Comment Actions
Tha faulty card replaced and at 2019-06-25 05:41 UTC the circuit recovered and running at the moment , please check and let us know if you have any issue .
Thanks for your patience and understanding
All back to normal.