Page MenuHomePhabricator
Create Task
Maniphest T315053

Enable OSPF Icinga check for EVPN based switches
Closed, ResolvedPublic
Actions

Description

A link between two of the EVPN-enabled switches in Eqiad went down recently, but we did not receive an alert.

I made an error in thinking that BGP would go down if a link failed, but given we're using iBGP for the EVPN SAFI, multihop is enabled which keeps the session up if the direct link is down. We may wish to look at this, changing the multihop attribute to force iBGP down if devices aren't directly connected.

But overall it would help to also alert on OSPF enabled interfaces vs OSPF adjacencies. This check exists for core routers already so we should be able to re-purpose the same one.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Enable OSPF check by default for l3 switch mgmt interfacesoperations/puppetproduction+1 -1
Adjust OSPF Icinga check to ignore OSPFv3 if zero ints configuredoperations/puppetproduction+5 -4
Customize query in gerrit

Event Timeline

cmooney created this task.Aug 12 2022, 12:21 AM
cmooney triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2022, 12:21 AM
cmooney added a comment.Aug 12 2022, 10:12 AM

Having thought about it in more detail I think it's best to keep the multihop for the iBGP EVPN sessions.

Reason being that even if a Leaf loses a Spine link it will still have two logical RR sessions. Which mean in case of an issue or config change each can be cleared separately, without removing all routes / disrupting traffic.

The OSPF adjacency check should catch failing ports, if not we can build another check for that.

cmooney added a subscriber: ayounsi.Aug 12 2022, 10:12 AM

@ayounsi be interested if you've any thoughts on that.

RhinosF1 subscribed.Aug 12 2022, 10:14 AM
ayounsi added a comment.Aug 12 2022, 12:17 PM

yeah I agree +1 on having a stable iBGP capable of handling link failure.

The OSPF adjacency check should be used but IIRC it assumes there are as many v4 sessions than v6 as that's what we have on the transport links. So a flag like --nov6 might be required.

• Donogee2021 closed this task as a duplicate of T315166: Trash.Aug 13 2022, 5:54 PM
JJMC89 reopened this task as Open.Aug 13 2022, 6:00 PM
• lmata added a project: Observability-Alerting.Aug 17 2022, 1:59 PM
• lmata moved this task from Inbox to Radar on the Observability-Alerting board.
gerritbot added a comment.Mar 15 2023, 12:52 PM

Change 899609 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Adjust OSPF Icinga check to ignore OSPFv3 if zero ints configured

https://gerrit.wikimedia.org/r/899609

gerritbot added a project: Patch-For-Review.Mar 15 2023, 12:52 PM
gerritbot added a comment.Mar 16 2023, 1:59 PM

Change 899609 merged by Cathal Mooney:

[operations/puppet@production] Adjust OSPF Icinga check to ignore OSPFv3 if zero ints configured

https://gerrit.wikimedia.org/r/899609

Maintenance_bot removed a project: Patch-For-Review.Mar 16 2023, 2:11 PM
gerritbot added a comment.Mar 16 2023, 7:51 PM

Change 900431 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] Enable OSPF check by default for l3 switch mgmt interfaces

https://gerrit.wikimedia.org/r/900431

gerritbot added a project: Patch-For-Review.Mar 16 2023, 7:51 PM
gerritbot added a comment.Mar 22 2023, 4:36 PM

Change 900431 merged by Cathal Mooney:

[operations/puppet@production] Enable OSPF check by default for l3 switch mgmt interfaces

https://gerrit.wikimedia.org/r/900431

cmooney closed this task as Resolved.Mar 22 2023, 5:09 PM

I've merged the patch and the EVPN switches are now being checked by Icinga, all looks healthy.

Maintenance_bot removed a project: Patch-For-Review.Mar 22 2023, 5:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits