Below automated task was created on the back of errors detected on the link. Further examination (see first comment) reveals the circuit is bouncing up and down frequently, optic reports -40dBm light regularly.
The link has been drained and we are following up with the carrier.
Common information
- dashboard: https://grafana.wikimedia.org/d/5p97dAASz/queue-and-error-stats-by-network-device?var-site=eqiad+prometheus%2Fops&var-device=cr2-eqiad&var-interface=et-1%2F1%2F5
- description:
- runbook: https://wikitech.wikimedia.org/wiki/Network_monitoring#Inbound/outbound_interface_errors
- summary: Outbound errors on interface cr2-eqiad:et-1/1/5 (Transport: cr2-codfw:et-0/1/4 (Lumen, 449169461))
- alertname: OutboundInterfaceErrors
- instance: cr2-eqiad:9804
- interface_description: Transport: cr2-codfw:et-0/1/4 (Lumen, 449169461)
- interface_name: et-1/1/5
- job: gnmi
- prometheus: ops
- scope: network
- severity: task
- site: eqiad
- source: prometheus
- team: dcops
Firing alerts
- dashboard: https://grafana.wikimedia.org/d/5p97dAASz/queue-and-error-stats-by-network-device?var-site=eqiad+prometheus%2Fops&var-device=cr2-eqiad&var-interface=et-1%2F1%2F5
- description:
- runbook: https://wikitech.wikimedia.org/wiki/Network_monitoring#Inbound/outbound_interface_errors
- summary: Outbound errors on interface cr2-eqiad:et-1/1/5 (Transport: cr2-codfw:et-0/1/4 (Lumen, 449169461))
- alertname: OutboundInterfaceErrors
- instance: cr2-eqiad:9804
- interface_description: Transport: cr2-codfw:et-0/1/4 (Lumen, 449169461)
- interface_name: et-1/1/5
- job: gnmi
- prometheus: ops
- scope: network
- severity: task
- site: eqiad
- source: prometheus
- team: dcops
- Source




