There is the (possibly very very minor) upside of the current situation that the very first rule in INPUT is

1018219 3110598857 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED

meaning all subsequent packets for a TCP flow after the SYNC don't need to traverse the INPUT chain at all, they get accepted immediately.

Just to understand this better, is this just about making $notrack default to true in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/ferm/manifests/service.pp#19 ? Cause I 'd +1 that.

In T240495#5734771, @akosiaris wrote:

Just to understand this better, is this just about making $notrack default to true in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/ferm/manifests/service.pp#19 ? Cause I 'd +1 that.

Agreed. (initially notrack wasn't part of the ferm::service definition and we had notrack exceptions via custom ferm::rule only)

Yes, it's about that $notrack default. My hypothesis is that setting it to true wouldn't break any traffic, wouldn't change the security situation much, but would eliminate a bunch of potential for conntrack table size issues when various services get overwhelmed. Some thoughts about why that hypothesis might be false:

• "Wouldn't change the security situation much" - Technically, the stateful tracking of the inbound conns does probably prevent some kinds of theoretical TCP flags attacks sorts of scenarios (some have existed in the past). But since these rules are for the destination host itself (as opposed to e.g. a firewall host protecting other hosts with other kernels and/or OSes), it would be weird and unlikely, I think, for the conntrack to prevent some invalid-flags scenario because it's dangerous while the same kernel's TCP is subject to some exploit or DoS from the same packet? In any case, we'd be using notrack for heavy public ports anyways (e.g. public authdns), so we don't have much choice in the most-important cases.

• There could be something I'm not thinking about, in the general case, for how notracking an inbound service port (the 2x raw rules it creates) causes an issue for the tracking related outbound connections? I don't think so, but I'm really not 100% sure here. DNS recursors and NTP servers (both of which exist for my example cases I'm looking at) give pause, since they both need to make tracked outbound connections to the outside world, and also allow inbound service to the same ports from only our networks:
  • DNS Recursor: Allows inbound service to udp+tcp port 53 from e.g. $PRODUCTION_NETWORKS, but also makes outbound tracked connections to random public hosts, where the remote destination port is 53.
  • NTP Server: Is similar in nature but even a little odder, as it's allowing inbound service to udp 123 for $PRODUCTION_NETWORKS, and also currently tracking outbound "connections" to remote hosts on port 123, but I believe it uses local port 123 as its source address for those connections.
  • Do scenarios like these break the general case that adding notrack to the inbound service port is functionally-ok? Or will it break half the tracking and cause the outbound connections to be broken?

In T240495#5735553, @BBlack wrote:

Yes, it's about that $notrack default. My hypothesis is that setting it to true wouldn't break any traffic, wouldn't change the security situation much, but would eliminate a bunch of potential for conntrack table size issues when various services get overwhelmed. Some thoughts about why that hypothesis might be false:

• "Wouldn't change the security situation much" - Technically, the stateful tracking of the inbound conns does probably prevent some kinds of theoretical TCP flags attacks sorts of scenarios (some have existed in the past). But since these rules are for the destination host itself (as opposed to e.g. a firewall host protecting other hosts with other kernels and/or OSes), it would be weird and unlikely, I think, for the conntrack to prevent some invalid-flags scenario because it's dangerous while the same kernel's TCP is subject to some exploit or DoS from the same packet? In any case, we'd be using notrack for heavy public ports anyways (e.g. public authdns), so we don't have much choice in the most-important cases.

For the non public facing things it would probably not change things much indeed, for the public ones, given that saturation based attacks are exceedingly easy these days I would be more worried about those than low rate weird TCP flag attacks/combinations being an attack vector that the statefulness of the firewall protects us from.

• There could be something I'm not thinking about, in the general case, for how notracking an inbound service port (the 2x raw rules it creates) causes an issue for the tracking related outbound connections? I don't think so, but I'm really not 100% sure here. DNS recursors and NTP servers (both of which exist for my example cases I'm looking at) give pause, since they both need to make tracked outbound connections to the outside world, and also allow inbound service to the same ports from only our networks:
  • DNS Recursor: Allows inbound service to udp+tcp port 53 from e.g. $PRODUCTION_NETWORKS, but also makes outbound tracked connections to random public hosts, where the remote destination port is 53.

That's fine. The outbound tracked connections originate from ephemeral ports and not from port 53 (but to dport 53), so neither the PREROUTING chain (which is checked on packets entering the machine) nor the OUTPUT chain (which is checked for packets originating from the machine) of the &NO_TRACK macro[1] that.

• NTP Server: Is similar in nature but even a little odder, as it's allowing inbound service to udp 123 for $PRODUCTION_NETWORKS, and also currently tracking outbound "connections" to remote hosts on port 123, but I believe it uses local port 123 as its source address for those connections.

That's a bit more convoluted so let's break it down:

• systemd-timesyncd (which is ran on the entire fleet minus the actual ntp servers) doesn't do that. It uses ephemeral ports as well and does not offer NTP services, so whatever applies to the above applies here as well. I just tested that on bast3004 for good measure with a tcpdump and a sudo timedatectl set-ntp false/true set. This isn't what we are discussing here, just adding it for completeness

• Various clients (e.g. network devices) ask our ntp servers using source port 123. This is fine again as these are incoming packets and even if they aren't tracked they are going to be accepted by the corresponding rule. The outgoing responses are going to be accepted anyway because OUTPUT has ACCEPT policy. Again, probably not what we are discussing here, adding it for completeness.

• The ntp servers synchronize between themselves (either ones in our control or just 3rd party ones), using dport and sport 123. So the NOTRACK[1] ferm macro rules would indeed match here and those connection would not be tracked. Inbound connections from ntp servers in our control (essentially $DOMAIN_NETWORKS) fall in the same category as various clients above. Again, no issue. Outbound connections from the ntp servers to 3rd party ones would not work however. The inbound packets from the 3rd party hosts would match no rule in iptables and hence be dropped. So this is the 1 case were we need to set notrack=false explicitly before we flip the default.

• Do scenarios like these break the general case that adding notrack to the inbound service port is functionally-ok? Or will it break half the tracking and cause the outbound connections to be broken?

I think we have 1 scenario up to now that would break and can treat it as the exception to the default. I guess we need to figure out if we have more? Perhaps we could use a proper NFLOG like rule into &NOTRACK (with rate limiting in the spirit of in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/manifests/base/firewall/log.pp#19) in order to catch any edge case we haven't thought about?

[1] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/ferm/files/functions.conf#26