It's somehow missing, which mean a change that is supposed to be brief (like a tcp-mss change) is causing the BGP sessions to fully go down, then back up. Which cause 2 BGP re-convergences in a short amount of time, which caused unreachability for the users going through those paths.
And audit all routers to make sure they have graceful-restart configured.
Steps are:
Will do this tonight, at or after 00:00 UTC.
Currently there are just a few (AMSIX peer) BGP sessions down:
cdanis@re0.cr2-esams> show bgp summary | match "(Active|Idle|Connect)" Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 80.249.208.180 9031 0 0 0 2 1w3d 5:44:07 Active 80.249.210.246 28598 8043 5166 0 3 1w4d 5:56:04 Idle 2001:7f8:1::a500:9031:1 9031 0 0 0 1 1w3d 5:44:07 Active
Will check for the same set after the maintenance.
Change 577363 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/dns@master] depool esams for cr2 router maintenance
Change 577363 merged by CDanis:
[operations/dns@master] depool esams for cr2 router maintenance
Mentioned in SAL (#wikimedia-operations) [2020-03-06T00:02:54Z] <cdanis> T246338 depool esams for router maintenance
Committed configuration at 00:21 UTC.
Took a few minutes for all BGP sessions to be recreated but eventually wound up with this:
cdanis@re0.cr2-esams> show bgp summary | match "(Active|Idle|Connect)" Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 80.249.208.180 9031 0 0 0 0 7:42 Active 80.249.209.236 1267 0 0 0 0 7:42 Connect 2001:7f8:1::a500:1267:2 1267 0 0 0 0 7:42 Active 2001:7f8:1::a500:9031:1 9031 0 0 0 0 7:42 Active
The session that was Idle is now established again, we just have one more AMS-IX peer (AS1267, WIND Telecomunicazioni S.p.A) that isn't working for whatever reason.
GracefulRestart shows up in all neighbor options.
Calling this a success
Change 577390 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/dns@master] Revert "depool esams for cr2 router maintenance"
Change 577390 merged by CDanis:
[operations/dns@master] Revert "depool esams for cr2 router maintenance"
Mentioned in SAL (#wikimedia-operations) [2020-03-06T00:33:58Z] <cdanis> repool esams T246338
cdanis@re0.cr2-esams> show bgp summary | match "(Active|Idle|Connect)" Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped... 80.249.208.180 9031 0 0 0 0 1:54:33 Active 2001:7f8:1::a500:9031:1 9031 0 0 0 0 1:54:33 Active
Discussion of rolling out graceful-restart to other dual-RE routers is at T191667#5948038