please let us know if we can assist :-)

Change #1269983 had a related patch set uploaded (by Blake; author: Blake):

[operations/alerts@master] kubernetes-generic: Add alerts for BGP failure scenarios.

https://gerrit.wikimedia.org/r/1269983

Change #1269994 had a related patch set uploaded (by Blake; author: Blake):

[operations/alerts@master] kubernetes-generic: Add alerts for BGP failure scenarios.

https://gerrit.wikimedia.org/r/1269994

Change #1269983 abandoned by Blake:

[operations/alerts@master] kubernetes-generic: Add alerts for BGP failure scenarios.

https://gerrit.wikimedia.org/r/1269983

Broadly the patch submitted looked good to me, though I see it was abandoned.

As per the comment I left on it we also need to alert when the session does not establish because the router/switch has not been configured for it. We could possibly detect that because there are no switch-exposed metrics for the given host. If we can't negatively alert on the series being missing we'd need to export the stats from Calico on the hosts instead. Unfortunately it doesn't do that as is, we'd need to instrument it ourselves but I think it's not too hard.

cmooney@wikikube-worker1273:~$ sudo calicoctl node status | awk '/node specific/{print "calico_bgp_peer_status=peer="$2", status="$11}'
calico_bgp_peer_status=peer=10.64.177.1, status=Established
calico_bgp_peer_status=peer=2620:0:861:137::1, status=Established

Ah, thanks Cathal! The original patch was abandoned because I was struggling with git, the new patch is now https://gerrit.wikimedia.org/r/c/operations/alerts/+/1269994. I'll update it in accordance with the comment on the previous patch.

In T356877#11808865, @cmooney wrote:

Broadly the patch submitted looked good to me, though I see it was abandoned.

As per the comment I left on it we also need to alert when the session does not establish because the router/switch has not been configured for it. We could possibly detect that because there are no switch-exposed metrics for the given host. If we can't negatively alert on the series being missing we'd need to export the stats from Calico on the hosts instead. Unfortunately it doesn't do that as is, we'd need to instrument it ourselves but I think it's not too hard.

cmooney@wikikube-worker1273:~$ sudo calicoctl node status | awk '/node specific/{print "calico_bgp_peer_status=peer="$2", status="$11}'
calico_bgp_peer_status=peer=10.64.177.1, status=Established
calico_bgp_peer_status=peer=2620:0:861:137::1, status=Established

We can also run the prometheus bird exporter as a sidecar to calico-node (container image needs to be build and some yaml needs to be edited) which should give us more standardized metrics. We could add that as a stretch goal for this task.

In T356877#11814024, @JMeybohm wrote:

We can also run the prometheus bird exporter as a sidecar to calico-node (container image needs to be build and some yaml needs to be edited) which should give us more standardized metrics. We could add that as a stretch goal for this task.

Yeah that's a better way to go if it works.

I gave a quick review on the CR, but you should at least use https://wikitech.wikimedia.org/wiki/Network_telemetry#remote_instance:gnmi_bgp_neighbor_session_state%7B%7D

If you also need the count of prefixes received by the switch we should implement the same "remote_instance" thing. That way you could also compare it to what's being sent on the host's side (for example with prometheus-bird-exporter)

Regarding the count of "prefixes received by the switch but not accepted", I think as a first step it could be a global Netops alert (for our internal peers).
However if it gets triggered more often by server side issues, we should move it to the same "remote_instance" mechanism.

Regarding the count of "prefixes received by the switch but not accepted", I think as a first step it could be a global Netops alert (for our internal peers).
However if it gets triggered more often by server side issues, we should move it to the same "remote_instance" mechanism.

I've open T423384: Investigate internal rejected prefixes for that.

Change #1269994 merged by jenkins-bot:

[operations/alerts@master] kubernetes-generic: Add alerts for BGP failure scenarios.

https://gerrit.wikimedia.org/r/1269994