Maniphest T431836

NodeBGPSessionStatusNotEstablished
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jul 10 2026, 1:36 PM

Project Tags

Referenced Files

None

Subscribers

Description

Common information

runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1054 has a BGP session which is not in the 'established' state.

alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1054:0
job: gnmi
peer_as: 64601
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
severity: task
site: eqiad
source: prometheus
team: serviceops

Firing alerts

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=lsw1-c6-eqiad&var-bgp_group=k8s4&var-bgp_neighbor=wikikube-worker1054&refresh=30s
description: Kubernetes node wikikube-worker1054 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (lsw1-c6-eqiad).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1054 has a BGP session which is not in the 'established' state.

address: 10.64.173.11
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1054:0
job: gnmi
network_instance_name: PRODUCTION
peer_as: 64601
peer_group: k8s4
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: BGP
remote_instance: lsw1-c6-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr2-eqiad&var-bgp_group=Kubernetes4&var-bgp_neighbor=wikikube-worker1054&refresh=30s
description: Kubernetes node wikikube-worker1054 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr2-eqiad).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1054 has a BGP session which is not in the 'established' state.

address: 10.64.32.49
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1054:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes4
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr2-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr2-eqiad&var-bgp_group=Kubernetes6&var-bgp_neighbor=wikikube-worker1054&refresh=30s
description: Kubernetes node wikikube-worker1054 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr2-eqiad).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1054 has a BGP session which is not in the 'established' state.

address: 2620:0:861:103:10:64:32:49
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1054:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes6
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr2-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=lsw1-c6-eqiad&var-bgp_group=k8s6&var-bgp_neighbor=wikikube-worker1054&refresh=30s
description: Kubernetes node wikikube-worker1054 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (lsw1-c6-eqiad).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1054 has a BGP session which is not in the 'established' state.

address: 2620:0:861:133:10:64:173:11
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1054:0
job: gnmi
network_instance_name: PRODUCTION
peer_as: 64601
peer_group: k8s6
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: BGP
remote_instance: lsw1-c6-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

Related Objects

Mentioned Here: T430290: Silence NodeBGPSessionStatusNotEstablished during reimages
T431443: NodeBGPSessionStatusNotEstablished

Event Timeline

phaultfinder created this task.Jul 10 2026, 1:36 PM

Restricted Application added a project: ServiceOps. · View Herald TranscriptJul 10 2026, 1:36 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

sre.k8s.renumber-node run between 12:56 (https://sal.toolforge.org/log/crgZTJ8BffdvpiTrxHtW) and 14:51 (https://sal.toolforge.org/log/WLiCTJ8BffdvpiTrvKe3) today.

Alert is no longer firing, and the BGP session with lsw1-c6-eqiad looks happy since at least 13:55:41, which is indeed ~ 20m after the alert fired:

$ sudo calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+---------------+-------+----------+-------------+
| PEER ADDRESS |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+--------------+---------------+-------+----------+-------------+
| 10.64.173.1  | node specific | up    | 13:55:41 | Established |
+--------------+---------------+-------+----------+-------------+

IPv6 BGP status
+-------------------+---------------+-------+----------+-------------+
|   PEER ADDRESS    |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+-------------------+---------------+-------+----------+-------------+
| 2620:0:861:133::1 | node specific | up    | 13:55:41 | Established |
+-------------------+---------------+-------+----------+-------------+

Looking at the cookbook logs, some interesting excerpts:

2026-07-10 12:57:28,351 blake 3397387 [INFO] Created silence ID ee263ea9-5159-4e73-96e2-025a9d0becb8 for 4:00:00
...
2026-07-10 13:33:46,903 blake 3397387 [ERROR] 100.0% (1/1) of nodes timeout while executing command #1: 'run-puppet-agent...et --attempts 60': alert1002.wikimedia.org
...
2026-07-10 13:33:46,997 blake 3397387 [INFO] END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1054.eqiad.wmnet with reason: host reimage
...
2026-07-10 13:33:47,014 blake 3397387 [INFO] Deleted silence ID ee263ea9-5159-4e73-96e2-025a9d0becb8

So, it seems there's this "handoff" of the silences between different phases of the cookbook, and it seems the second one failed at the alert1002.wikimedia.org puppet-run stage, presumably due to the ongoing puppetserver weirdness.

Anyway, given the timings, and the fact that the flapping BGP session on cr2-eqiad was cleared by around ~ 14:35 (grafana), I think this all would have been fine, had the second silence succeeded.

See discussion in T431443 and T430290.

cc: @Blake FYI in case you think there's anything worth looking at after-the-fact here.

I don't think there's any follow-up remaining here, and agree that, had the silence succeeded for the new host, everything would have been fine. Thanks, Scott!