NodeBGPSessionStatusNotEstablished
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jun 25 2026, 7:11 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

MLechvien-WMF

phaultfinder

Description

Common information

runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished

alertname: NodeBGPSessionStatusNotEstablished
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
severity: task
site: eqiad
source: prometheus
team: serviceops

Firing alerts

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Kubernetes4&var-bgp_neighbor=wikikube-worker1162:0&refresh=30s
description: Kubernetes node wikikube-worker1162:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr1-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1162:0 has a BGP session which is not in the 'established' state.
address: 10.64.48.108
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1162:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes4
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr1-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr2-eqiad:9804&var-bgp_group=Kubernetes4&var-bgp_neighbor=wikikube-worker1162:0&refresh=30s
description: Kubernetes node wikikube-worker1162:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr2-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1162:0 has a BGP session which is not in the 'established' state.
address: 10.64.48.108
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1162:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes4
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr2-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Kubernetes4&var-bgp_neighbor=wikikube-worker1160:0&refresh=30s
description: Kubernetes node wikikube-worker1160:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr1-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1160:0 has a BGP session which is not in the 'established' state.
address: 10.64.48.116
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1160:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes4
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr1-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr2-eqiad:9804&var-bgp_group=Kubernetes4&var-bgp_neighbor=wikikube-worker1160:0&refresh=30s
description: Kubernetes node wikikube-worker1160:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr2-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1160:0 has a BGP session which is not in the 'established' state.
address: 10.64.48.116
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1160:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes4
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr2-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Kubernetes4&var-bgp_neighbor=wikikube-worker1161:0&refresh=30s
description: Kubernetes node wikikube-worker1161:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr1-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1161:0 has a BGP session which is not in the 'established' state.
address: 10.64.48.118
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1161:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes4
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr1-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr2-eqiad:9804&var-bgp_group=Kubernetes4&var-bgp_neighbor=wikikube-worker1161:0&refresh=30s
description: Kubernetes node wikikube-worker1161:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr2-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1161:0 has a BGP session which is not in the 'established' state.
address: 10.64.48.118
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1161:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes4
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr2-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Kubernetes6&var-bgp_neighbor=wikikube-worker1162:0&refresh=30s
description: Kubernetes node wikikube-worker1162:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr1-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1162:0 has a BGP session which is not in the 'established' state.
address: 2620:0:861:107:10:64:48:108
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1162:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes6
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr1-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr2-eqiad:9804&var-bgp_group=Kubernetes6&var-bgp_neighbor=wikikube-worker1162:0&refresh=30s
description: Kubernetes node wikikube-worker1162:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr2-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1162:0 has a BGP session which is not in the 'established' state.
address: 2620:0:861:107:10:64:48:108
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1162:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes6
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr2-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Kubernetes6&var-bgp_neighbor=wikikube-worker1160:0&refresh=30s
description: Kubernetes node wikikube-worker1160:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr1-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1160:0 has a BGP session which is not in the 'established' state.
address: 2620:0:861:107:10:64:48:116
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1160:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes6
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr1-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr2-eqiad:9804&var-bgp_group=Kubernetes6&var-bgp_neighbor=wikikube-worker1160:0&refresh=30s
description: Kubernetes node wikikube-worker1160:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr2-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1160:0 has a BGP session which is not in the 'established' state.
address: 2620:0:861:107:10:64:48:116
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1160:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes6
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr2-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Kubernetes6&var-bgp_neighbor=wikikube-worker1161:0&refresh=30s
description: Kubernetes node wikikube-worker1161:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr1-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1161:0 has a BGP session which is not in the 'established' state.
address: 2620:0:861:107:10:64:48:118
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1161:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes6
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr1-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

dashboard: https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbor-detail-by-network-device?orgId=1&from=now-30m&to=now&timezone=utc&var-site=eqiad&var-device=cr2-eqiad:9804&var-bgp_group=Kubernetes6&var-bgp_neighbor=wikikube-worker1161:0&refresh=30s
description: Kubernetes node wikikube-worker1161:0 has a BGP session which is not in the 'established' state. The node will not be able to establish connectivity to its switch (cr2-eqiad:9804).
runbook: https://wikitech.wikimedia.org/wiki/Kubernetes/Administration#NodeBGPSessionStatusNotEstablished
summary: Kubernetes node wikikube-worker1161:0 has a BGP session which is not in the 'established' state.
address: 2620:0:861:107:10:64:48:118
alertname: NodeBGPSessionStatusNotEstablished
instance: wikikube-worker1161:0
job: gnmi
network_instance_name: DEFAULT
peer_as: 64601
peer_group: Kubernetes6
peer_type: EXTERNAL
prometheus: ops
protocol_identifier: BGP
protocol_name: DEFAULT
remote_instance: cr2-eqiad:9804
severity: task
site: eqiad
source: prometheus
team: serviceops
Source

Related Objects

Mentioned In: T430290: Silence NodeBGPSessionStatusNotEstablished during reimages
Mentioned Here: T421711: ServiceOps: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets
T430226: Automate workflow for vlan migrations on k8s worker nodes

Event Timeline

phaultfinder created this task.Jun 25 2026, 7:11 PM

Restricted Application added a project: ServiceOps. · View Herald TranscriptJun 25 2026, 7:11 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

phaultfinder updated the task description. (Show Details)Jun 25 2026, 8:21 PM

phaultfinder updated the task description. (Show Details)Jun 25 2026, 9:31 PM

Assigning to Jasmine to take a look

Blake mentioned this in T430290: Silence NodeBGPSessionStatusNotEstablished during reimages.Jun 26 2026, 10:17 AM

Resolving as this is an artifact from vlan migrations work in T421711: ServiceOps: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets.

A brief summary is that we do not yet have a workflow for vlan migrations on worker nodes. The reimage cookbook which supports vlan migrations sre.hosts.reimage isn’t designed for worker nodes since it’s missing drain and cordon logic and sre.k8s.roll-reimage-node (the cookbook that is designed for worker node reimages) doesn’t support vlan migrations at current either.

These alerts fired as a result of reimaging the worker nodes with the sre.hosts.reimage cookbook, which does not support drain/cordon logic nor handle BGP.

The temporary work around is to use the pool-depool cookbook [0] which handles BGP and would avoid this alert firing in the future. The more long term solution is to either add support for vlan migrations in sre.k8s.roll-reimage-nodes or write a separate cookbook for this work altogether.

Further details in T430226: Automate workflow for vlan migrations on k8s worker nodes

[0] - https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/k8s/pool-depool-node.py

NodeBGPSessionStatusNotEstablishedClosed, ResolvedPublicActions