Page MenuHomePhabricator

cr1/2-eqiad PFE_FW_SYSLOG_IP6_GEN log entries
Closed, ResolvedPublic

Description

While investigating network issues, I was looking into cr1/cr2-eqiad logs and found quite a lot of logs e.g.:

Aug  2 16:45:11  re0.cr1-eqiad fpc3 PFE_FW_SYSLOG_IP6_GEN: FW: ae3.1022     D 0 SA fe80:0:0:0:d294:66ff:fe1b:3367  DA ff02:0:0:0:0:0:0:202  (1 packets)
Aug  2 16:45:28  re0.cr1-eqiad fpc3 PFE_FW_SYSLOG_IP6_GEN: FW: ae3.1022     D 0 SA fe80:0:0:0:d294:66ff:fe1b:3367  DA ff02:0:0:0:0:0:0:202  (1 packets)
Aug  2 16:46:17  re0.cr1-eqiad fpc3 PFE_FW_SYSLOG_IP6_GEN: FW: ae2.1021     D 0 SA fe80:0:0:0:d294:66ff:fe1b:49a6  DA ff02:0:0:0:0:0:0:202  (1 packets)
Aug  2 16:46:39  re0.cr1-eqiad fpc3 PFE_FW_SYSLOG_IP6_GEN: FW: ae2.1021     D 0 SA fe80:0:0:0:d294:66ff:fe1b:49a6  DA ff02:0:0:0:0:0:0:202  (1 packets)
Aug  2 16:47:19  re0.cr1-eqiad fpc3 PFE_FW_SYSLOG_IP6_GEN: FW: ae3.1022     D 0 SA fe80:0:0:0:d294:66ff:fe1b:3367  DA ff02:0:0:0:0:0:0:202  (1 packets)
Aug  2 16:47:28  re0.cr1-eqiad fpc3 PFE_FW_SYSLOG_IP6_GEN: FW: ae3.1022     D 0 SA fe80:0:0:0:d294:66ff:fe1b:3367  DA ff02:0:0:0:0:0:0:202  (1 packets)
Aug  2 16:48:26  re0.cr1-eqiad fpc3 PFE_FW_SYSLOG_IP6_GEN: FW: ae2.1021     D 0 SA fe80:0:0:0:d294:66ff:fe1b:49a6  DA ff02:0:0:0:0:0:0:202  (1 packets)
Aug  2 16:48:50  re0.cr1-eqiad fpc3 PFE_FW_SYSLOG_IP6_GEN: FW: ae2.1021     D 0 SA fe80:0:0:0:d294:66ff:fe1b:49a6  DA ff02:0:0:0:0:0:0:202  (1 packets)

…seems to be repeated every few minutes.

There are also these:

Aug  2 16:51:14  re0.cr1-eqiad /kernel: Nexthop index allocation failed: private index space exhausted
Aug  2 16:51:15  re0.cr1-eqiad /kernel: Nexthop index allocation failed: private index space exhausted

and

Aug  2 18:36:16  re0.cr1-eqiad fpc3 Next-hop resolution requests from interface 329 throttled
Aug  2 18:57:03  re0.cr1-eqiad fpc3 Next-hop resolution requests from interface 418 throttled
Aug  2 19:16:02  re0.cr1-eqiad fpc4 Next-hop resolution requests from interface 414 throttled

which are probably (but not certainly) related, and a bit more worrying.

At minimum, this is noise that should be silenced; at most, this is an operational issue we need to resolve.

Event Timeline

faidon triaged this task as High priority.Aug 3 2018, 9:14 AM
faidon created this task.

/kernel: Nexthop index allocation failed due to router limitation and the design of our mgmt network (see description of T174397)

PFE_FW_SYSLOG_IP6_GEN is temporary logging to help with: T198623, those specific ff02::202 messages are due to rpcbind, (see T198623#4423310 )

Next-hop resolution requests seems to a a non issue https://forums.juniper.net/t5/Junos/Next-hop-resolution-requests-from-interface/m-p/321843#M12732

I disabled logging a couple weeks ago, so no more PFE_FW_SYSLOG_IP6_GEN logs.

Next-hop resolution requests from interface XXX throttled are probably due to devices port/ping scanning our whole public ranges. this could be investigated but the fix might be to have tighter firewall filters (only allow traffic to IPs that have servers listening on them. But the cost of fixing might be higher than the benefits (as it's not causing production issues),

At best we could hide the rules with "match" rules in the syslog configuration, but I think it's better to have them rather than hide them (as long as the noise level stays low)

Closing that task, feel free to reopen if needed.