While rolling out the updated loopback filter in drmrs I discovered I could not display the status of NTP peers on asw1-b12-drmrs:
cmooney@asw1-b12-drmrs> show ntp associations no-resolve localhost: timed out, nothing received ***Request timed out
Knowing in the backgrround this is probably just running "ntpq -p", which connects to the local NTP daemon on port 123, I assumed the modifications to the loopback filter (T304553) had caused the issue. However I was unable to work out what was wrong, a 'monitor traffic' on the loopback interface does show the connection going out, and indeed no response is received:
12:41:06.335039 In IP 185.15.58.131.63975 > 185.15.58.131.123: NTPv2, Reserved, length 12
The source and destination IP is the loopback address, which is part of 185.15.56.0/24, which is part of the "production4" prefix list the filter matches. So it should be allowed.
In tests the loopback filter did not seem to be causing the issue. Even with an "allow all" on the loopback filter, which I temporarily configured after some other tests (making it more permissive each time), the command still fails. Same with the filter removed completely from lo0.
The updated loopback filter is now applied to asw1-b13-drmrs, and the command runs fine there:
cmooney@asw1-b13-drmrs> show ntp associations no-resolve remote refid st t when poll reach delay offset jitter =============================================================================== +208.80.154.10 170.187.158.81 3 - 9 64 1 85.265 -0.475 0.031 -208.80.155.108 45.79.214.107 3 - 8 64 1 85.278 0.648 0.035 -208.80.153.77 38.229.56.9 3 - 7 64 1 116.169 0.193 0.054 +208.80.153.111 104.156.229.103 3 - 6 64 1 116.091 -0.191 0.862 +91.198.174.61 91.198.174.62 3 - 5 64 1 166.121 -0.400 0.044 *91.198.174.62 83.137.149.135 2 - 4 64 1 166.191 -0.423 0.031
Configuration on both devices, in terms of NTP, looks identical. So I'm a bit at a loss as to what to do. Potentially ntpd is in some crashed/bad state and if we restart/remove and re-add the config it will clear.
I don't think today's changes to the filter have affected the situation anyway, I think whatever it is was going on beforehand too. Will discuss in netops and decide what the best way forward is.