Page MenuHomePhabricator
Create Task
Maniphest T187994

netfilter software at WMF: iptables vs nftables
Closed, DeclinedPublic
Actions

Description

Here at the WMF, we use a lot Netfilter software, mostly iptables using the ferm wrapper (ops/puppet). Also other software stacks like kubernetes and openstack are using iptables to manage networking.
However, iptables is being replaced by nftables.

This new framework (well, not that new) brings some interesting features for us, which can drastically improve some of our use cases, and remove our needs for wrappers like ferm:

  • native dual stack support for IPv4 & IPv6
  • native support for performance structures, like sets, maps, dictionaries and concatenations
  • improved debugging and reporting options
  • native support for all classic families in a single tool (vs the split: iptables, ip6tables, arptables, ebtables)
  • ingress hook support for improved performance
  • flexible ruleset layout, vs the classical static layout of iptables (i.e. filter/INPUT, nat/PREROUTING, etc)
  • optional counters and multiple actions in a single rule (i.e, you can increment a counter, log and NAT in the same rule)
  • improved ruleset management options, fully atomic and incremental ruleset updates, improved error reporting, etc
  • all the ecosystem and tooling is supported (conntrack-tools, sysctl keys, libs, etc)

In our case (WMF), an eventual migration from iptables (ferm) to nftables would require:

  • a puppet rewrite of the affected modules (base::firewall and friends)
  • external software stacks integration with nftables (i.e, k8s and openstack generating nftables rulesets vs iptables)

External software project should address the migration by themselves, I don't plan to send patches unless we determine it's crucial or key for our business.
There is also a Netfilter upstream compat/translation effort in place to help in migrations, but I don't think we need those (well, we could trick k8s or openstack so they generate nftables rules without a single change in their codebase).
I maintain the Debian nftables packages, they in good shape and uptodate, ready to use.

This is, for example, an iptables ruleset generated by openstack in our WMCS infra, which could be rewritten natively in nftables using 2 or 3 rules with sets and maps.

# Generated by iptables-save vX.X.XX on Mon Feb XX XX:XX:XX XXXX
*filter
:INPUT ACCEPT XXXXXXXXXXX:XXXXXXXXXXXXXXX
:FORWARD ACCEPT XXXXXXXXXXX:XXXXXXXXXXXX
:OUTPUT ACCEPT XXXXXXXXXXX:XXXXXXXXXXXXXX
:nova-api-FORWARD - XX:XX
:nova-api-INPUT - XX:XX
:nova-api-OUTPUT - XX:XX
:nova-api-local - XX:XX
:nova-filter-top - XX:XX
:nova-network-FORWARD - XX:XX
:nova-network-INPUT - XX:XX
:nova-network-OUTPUT - XX:XX
:nova-network-local - XX:XX
-A INPUT -j nova-network-INPUT
-A INPUT -j nova-api-INPUT
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-network-FORWARD
-A FORWARD -j nova-api-FORWARD
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-network-OUTPUT
-A OUTPUT -j nova-api-OUTPUT
-A nova-api-INPUT -d XX.XX.XX.XX/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A nova-filter-top -j nova-network-local
-A nova-filter-top -j nova-api-local
-A nova-network-FORWARD -i brXXXX -j ACCEPT
-A nova-network-FORWARD -o brXXXX -j ACCEPT
-A nova-network-INPUT -i brXXXX -p udp -m udp --dport XX -j ACCEPT
-A nova-network-INPUT -i brXXXX -p tcp -m tcp --dport XX -j ACCEPT
-A nova-network-INPUT -i brXXXX -p udp -m udp --dport XX -j ACCEPT
-A nova-network-INPUT -i brXXXX -p tcp -m tcp --dport XX -j ACCEPT
COMMIT
# Completed on Mon Feb XX XX:XX:XX XXXX
# Generated by iptables-save vX.X.XX on Mon Feb XX XX:XX:XX XXXX
*mangle
:PREROUTING ACCEPT XXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXX
:INPUT ACCEPT XXXXXXXXXXX:XXXXXXXXXXXXXXX
:FORWARD ACCEPT XXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXX
:OUTPUT ACCEPT XXXXXXXXXXX:XXXXXXXXXXXXXX
:POSTROUTING ACCEPT XXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXX
:nova-api-POSTROUTING - XX:XX
:nova-network-POSTROUTING - XX:XX
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -j nova-api-POSTROUTING
-A nova-network-POSTROUTING -o brXXXX -p udp -m udp --dport XX -j CHECKSUM --checksum-fill
COMMIT
# Completed on Mon Feb XX XX:XX:XX XXXX
# Generated by iptables-save vX.X.XX on Mon Feb XX XX:XX:XX XXXX
*nat
:PREROUTING ACCEPT XXXXXXXXXXXX:XXXXXXXXXXXXXX
:INPUT ACCEPT XXXXXXXXXX:XXXXXXXXXXXX
:OUTPUT ACCEPT XXXXXXXX:XXXXXXXXXX
:POSTROUTING ACCEPT XXXXXXXXXXX:XXXXXXXXXXXX
:nova-api-OUTPUT - XX:XX
:nova-api-POSTROUTING - XX:XX
:nova-api-PREROUTING - XX:XX
:nova-api-float-snat - XX:XX
:nova-api-snat - XX:XX
:nova-network-OUTPUT - XX:XX
:nova-network-POSTROUTING - XX:XX
:nova-network-PREROUTING - XX:XX
:nova-network-float-snat - XX:XX
:nova-network-snat - XX:XX
:nova-postrouting-bottom - XX:XX
-A PREROUTING -j nova-network-PREROUTING
-A PREROUTING -j nova-api-PREROUTING
-A OUTPUT -j nova-network-OUTPUT
-A OUTPUT -j nova-api-OUTPUT
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -j nova-api-POSTROUTING
-A POSTROUTING -j nova-postrouting-bottom
-A nova-api-snat -j nova-api-float-snat
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -d XX.XX.XX.XX/XX -j ACCEPT
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -d XXX.XX.XXX.X/XX -j ACCEPT
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -d XX.X.X.X/X -j ACCEPT
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -m conntrack ! --ctstate DNAT -j ACCEPT
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-PREROUTING -d XXX.XXX.XXX.XXX/XX -p tcp -m tcp --dport XX -j DNAT --to-destination XX.XX.XX.XX:XXXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-snat -j nova-network-float-snat
-A nova-network-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-postrouting-bottom -j nova-network-snat
-A nova-postrouting-bottom -j nova-api-snat
COMMIT
# Completed on Mon Feb XX XX:XX:XX XXXX

I'm opening this ticket to collect comments and questions across teams which may have different point of views, use cases, tradeoffs and requirements for an eventual migration.

More info at http://wiki.nftables.org/ (pleny of docs and examples).

For those who may know nothing about nftables, this is what a ruleset in my laptop looks like:

⏚ arturo@endurance:~$ sudo nft list ruleset
table inet filter {
	chain input {
		type filter hook input priority 0; policy accept;
		iif "lo" accept
		ct state established,related accept
		icmpv6 type { nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
		counter packets 4990 bytes 472427 drop
	}
}

Related Objects

Event Timeline

aborrero created this task.Feb 22 2018, 1:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 22 2018, 1:00 PM
elukey subscribed.Feb 22 2018, 1:03 PM
faidon added a comment.Feb 22 2018, 1:36 PM

However, iptables is being replaced by nftables.

It seems to me like nftables is still not very widely used (as also evidenced by the upstreams you mentioned not having adopted it yet) and we might be early adopters; is that your impression as well?

What's your take on the bpfilter work, and emails like this one from Dave Miller? It sounds like iptables may do a comeback, but it's still debated upstream, right?

Is there feature parity for everything that we do or that we'll reasonably want to do with nftables? One of the comments on the LWN article seemed to suggest that e.g. MSS clamping (which we don't use at Wikimedia, but it was an example of an extremely common thing people do) wasn't implemented in nftables until Linux 4.14.

This new framework (well, not that new) brings some interesting features for us, which can drastically improve some of our use cases, and remove our needs for wrappers like ferm
<snip>

In practice, which of a) our workflows b) our use cases would benefit from such a migration? Things that we can't do today or that we do suboptimally, issues we've experienced that would be addressed, etc. The features you listed are pretty nice -and I've been personally been bitten by some of these limitations in the past!- but not very relevant to Wikimedia (dual-stacking we do with ferm, we don't use ebtables/arptables, we don't do NAT outside of OpenStack etc.)

MoritzMuehlenhoff subscribed.Feb 22 2018, 1:46 PM
aborrero added a comment.EditedFeb 22 2018, 2:31 PM
In T187994#3992672, @faidon wrote:

However, iptables is being replaced by nftables.

It seems to me like nftables is still not very widely used (as also evidenced by the upstreams you mentioned not having adopted it yet) and we might be early adopters; is that your impression as well?

I know of some people who are working hard to integrate upstream projects with nftables. Redhat with firewalld is one of them. I guess at some point other layered products like openshift and openstack will gain support for it as well.
The adoption is delayed because is always easier to stick to the tooling you know (see ifconfig vs ip). At some point, Debian itself will stop using iptables and only use nftables, natively or by means of the compat layer.

I don't think we are early adopters (what does that even mean). I know of other DC who migrated to nftables, I did myself the work in some cases (citation http://workshop.netfilter.org/2017/wiki/index.php/Developer_days.html#nftables_at_CICA.2C_our_experience).
You can't compare the numbers of iptables deployments vs nftables because iptables has been here since the 90'.

Also, at some point iptables will be in a 'maintenance only' mode. I always prefer using software which is being actively developed.

What's your take on the bpfilter work, and emails like this one from Dave Miller? It sounds like iptables may do a comeback, but it's still debated upstream, right?

The bpfilter push is based on addressing big silicon datacenters usecases which want to do offloading and/or use smartnics. It seems they have little interest in addressing common interests of wider audiences.
They want to keep the old iptables interfaces, which is perpetuating iptables mistakes and ignoring the already in place push for nftables. Also I would like to keep sysadmins away from having compiler suites installed in their firewall machines and having to deal with assembly code in case something needs to be debugged.

The Netfilter project doesn't discard adding eBPF support to nftables for the purpose of offloading usecases but that patchset was like sending to GNOME something like "[RFC PATCH] replace gnome with KDE".

Is there feature parity for everything that we do or that we'll reasonably want to do with nftables? One of the comments on the LWN article seemed to suggest that e.g. MSS clamping (which we don't use at Wikimedia, but it was an example of an extremely common thing people do) wasn't implemented in nftables until Linux 4.14.

I would say that everything we need is already covered by nftables. Some extensions have no equivalent in nftables and some others will be deprecated (i.e, wont be translated) but we are always talking about corner cases.
We couldn't develop every single equivalent since the beginning, that's a huge work, but a long road have been walked already.

This new framework (well, not that new) brings some interesting features for us, which can drastically improve some of our use cases, and remove our needs for wrappers like ferm
<snip>

In practice, which of a) our workflows b) our use cases would benefit from such a migration? Things that we can't do today or that we do suboptimally, issues we've experienced that would be addressed, etc. The features you listed are pretty nice -and I've been personally been bitten by some of these limitations in the past!- but not very relevant to Wikimedia (dual-stacking we do with ferm, we don't use ebtables/arptables, we don't do NAT outside of OpenStack etc.)

ferm is a great tool, but a wrapper after all. From my POV using native interfaces is always better than workarounds.

Right now our workflow would be similar, i.e, git, puppet, etc. This may change in the future, and if it changes, it would be better for us to be using a more flexible framework.

Our use cases could benefit from nftables in several aspects:

  • performance, by using sets, maps, dicts and concatenations instead of repeating rules. Even more important if k8s and openstack use nftables, i.e. cloud environments (see example ruleset above)
  • actual IPv4/IPv6 dual stack, ferm is just hiding the details behind a common syntax. The nftables engine uses real dual stacking in the kernel (the inet family)
  • load balancing, we could implement layer 2/3/4 load balancing natively with nftables, we could replace also LVS (citation: https://wiki.nftables.org/wiki-nftables/index.php/Load_balancing)

A simple example of the performance gain is to have a single rule:

ip daddr @allowed_addresses_set tcp dport 22 accept

And then add addresses to the set, instead of having many rules. This gain is even more important if we use maps, dictionaries and concatenations.

jcrespo subscribed.Feb 22 2018, 3:58 PM

I am agnostic about this, do not know enough differences- arguably, I think we should better puppetize the current rules- but I have never found problems with the backends to understand the details. But I do not run complex things like openstack or the network/traffic "stacks".

The only thing I would be commenting is that anything new should provide clear advantages to pay for the cost of the migration (rarely these things can be done fully online)- and the orchestration of certain layers is very lacking. For comparison, ferm implementation for me took me 2 years, and it is still in a very basic state.

aborrero added a comment.Feb 22 2018, 4:53 PM
In T187994#3992995, @jcrespo wrote:

I am agnostic about this, do not know enough differences- arguably, I think we should better puppetize the current rules- but I have never found problems with the backends to understand the details. But I do not run complex things like openstack or the network/traffic "stacks".

The only thing I would be commenting is that anything new should provide clear advantages to pay for the cost of the migration (rarely these things can be done fully online)- and the orchestration of certain layers is very lacking. For comparison, ferm implementation for me took me 2 years, and it is still in a very basic state.

Yes, a full migration in every corner of our infra could take us 1-2 years, at least. I guess is like upgrading other key pieces of the infra, like p3 to p4, or choosing a different base OS. But I believe that having ferm already in place is an advantage, since syntax is similar, layout is similar, etc.

I did complex migrations in the past because rather complex rulesets/environments and because the migration often comes with friends (f.e. let's change network topology or whatever).

aborrero updated the task description. (Show Details)Feb 22 2018, 5:20 PM
ayounsi subscribed.Feb 22 2018, 6:26 PM
bd808 subscribed.Feb 22 2018, 7:32 PM
Marostegui subscribed.Feb 23 2018, 9:12 AM
aborrero added a comment.Feb 26 2018, 10:02 AM
In T187994#3992756, @aborrero wrote:

[...]
Our use cases could benefit from nftables in several aspects:

  • performance, by using sets, maps, dicts and concatenations instead of repeating rules. Even more important if k8s and openstack use nftables, i.e. cloud environments (see example ruleset above)
  • actual IPv4/IPv6 dual stack, ferm is just hiding the details behind a common syntax. The nftables engine uses real dual stacking in the kernel (the inet family)
  • load balancing, we could implement layer 2/3/4 load balancing natively with nftables, we could replace also LVS (citation: https://wiki.nftables.org/wiki-nftables/index.php/Load_balancing)

Did I mention that nftables can outperform LVS by about 10x? (https://netdevconf.org/1.2/slides/oct6/08_nftables_Load_Balancing_with_nftables_II_Slides.pdf) This could translate to improved throughput and less money spent in hardware.

aborrero added subscribers: • ema, BBlack.Feb 26 2018, 10:04 AM
MoritzMuehlenhoff triaged this task as Medium priority.Feb 26 2018, 11:20 AM
Chicocvenancio subscribed.Feb 27 2018, 4:24 PM
faidon added a comment.Feb 28 2018, 3:23 AM

I don't think it's easy for anyone to calculate the amount of effort required for this, but the stated 1-2 year long migration sounds longer than I thought and... pretty scary. I'd like to at least be conscious of the amount of effort required here, and foresee clear, tangible benefits at the end of the line to be able to justify the effort both for the migration itself, plus all the associated risks, learning curve and confusion in the meantime.

The arguments you mentioned above are:

  • Performance in large rulesets; can you see any non-virt/container specific cases in our rulesets where we have such rulesets and where nftables would make a material difference? We typically don't even use subchains, due to the simplicity of our rulesets, but I haven't looked in a long while :)
  • OpenStack and k8s: we depend on upstream code for both; is the suggestion to patch either of these ourselves?
  • not having to use a wrapper and by not doing so, also not having to deal with IPv4/IPv6 separately. Given that we've had little to no issues with ferm in general and with dual stacking in particular, it's hard for me to see the practical benefit. Are there specific problems you have in mind?
  • load balancing/IPVS: the performance improvements sound pretty awesome, but note that the complexity of that migration would be much higher, given we'd need to rewrite a good chunk of Pybal. The monetary benefits would also be slim to non-existent, as CPU performance is not really the driver for our cluster sizes. It sounds to me like we have a lot to lose and little to gain in this case, unfortunately.

Anything else I'm missing?

aborrero added a comment.Mar 1 2018, 2:38 PM
In T187994#4009038, @faidon wrote:

I don't think it's easy for anyone to calculate the amount of effort required for this, but the stated 1-2 year long migration sounds longer than I thought and... pretty scary. I'd like to at least be conscious of the amount of effort required here, and foresee clear, tangible benefits at the end of the line to be able to justify the effort both for the migration itself, plus all the associated risks, learning curve and confusion in the meantime.

I would need help in calculating. This 1-2 year guesstimate could be strongly inaccurate. I could work in preliminary ops/puppet.git patches to get a better sense of this stuff.

The arguments you mentioned above are:

  • Performance in large rulesets; can you see any non-virt/container specific cases in our rulesets where we have such rulesets and where nftables would make a material difference? We typically don't even use subchains, due to the simplicity of our rulesets, but I haven't looked in a long while :)

Well, I believe performance is a thing.
Also, note that mixing iptables and nftables could be problematic, specially when doing NAT. Therefore we should aim for a complete migration, not only in virt/container environments.

I don't usually jump to many servers outside WCMS, but the several I know more could have great benefits:

aborrero@labcontrol1001:~$ sudo ip6tables-save  | wc -l
115
aborrero@labcontrol1001:~$ sudo iptables-save  | wc -l
172

These ~300 rules could be reduced to something like 5. These are plain filtering rules, not generated by openstack.

In case of doubts, this means: for every packet which enters the sever, evaluate just 5 rules.

I'm sure we could find other servers which have even bigger rulesets.

  • OpenStack and k8s: we depend on upstream code for both; is the suggestion to patch either of these ourselves?

No, but we could if we decide to do so. Apart of patching upstream code, a migration path could be to simply 'trick' them to use nftables by symlinking the binaries to the compat ones:

  • /usr/sbin/iptables --> /usr/sbin/iptables-compat
  • /usr/sbin/iptables-restore --> /usr/sbin/iptables-compat-restore
  • not having to use a wrapper and by not doing so, also not having to deal with IPv4/IPv6 separately. Given that we've had little to no issues with ferm in general and with dual stacking in particular, it's hard for me to see the practical benefit. Are there specific problems you have in mind?

At quick glance, some tasks which may benefit from using nftables rather than ferm, by means of different nftables features (several actions in a single rule, atomic operations, non-static ruleset architecture, etc):

  • load balancing/IPVS: the performance improvements sound pretty awesome, but note that the complexity of that migration would be much higher, given we'd need to rewrite a good chunk of Pybal. The monetary benefits would also be slim to non-existent, as CPU performance is not really the driver for our cluster sizes. It sounds to me like we have a lot to lose and little to gain in this case, unfortunately.

Surely, migrating to a new technology isn't free. But, a lot to lose? Could you elaborate?

BTW pybal/ipvs.py is ~250 LOC. I don't think this it's a big deal, at least in terms of LOCs.
There are people out there building load balancers with nftables ( https://github.com/zevenet/nftlb), which could inspire us when adding support for PyBal.

Anything else I'm missing?

One of the main points is moving to a newer technology itself, instead of keep using software from the 90'.
By using a modern technology we could ensure we are more ready for the future, use cases to come which we may not have currently.
At some point, I think Debian (and others distros, like RedHat) will switch to nftables by default and offer iptables as a legacy tool. Are we ready for it? In case of Debian, this may happen in Buster if I find the proper time to work on it.

faidon added a comment.Mar 1 2018, 3:31 PM

First, I don't think we should be thinking in terms of "using software from the 90s", at least not for something that is still as widely used and well-maintained as iptables (and to something that is as seldomly used as nftables). This is not something we should judge software with; we can talk instead in terms of amounts of bugs, maintainability, upstream response times, when was the last release, if/when it was deprecated by upstream(s) etc.

Second, aiming for better performance everywhere in our stack is a noble goal of course, but our engineering resources are finite and very limited, and we have to pick our battles wisely. One has to always wonder "for the equal amount of effort, what would make the biggest impact?". Do you feel that nftables would be either a quick/easy win, or alternatively, a huge win in terms of performance and maintainability? What kind of actual impact would that be?

It's possible that there _are_ use cases that we'd benefit hugely from, but none of the ones mentioned above seem to be compelling ones. For instance, in the labcontrol1001 case:

  • Are there *actual* performance problems on this box with its ~150 rules per afi? Do you feel like nftables there would substantially change any of the workloads or, alternatively, the amount or cost of those systems? If you were to migrate it tomorrow, what would be the benefit that we'd see at a higher level?
  • How much could we optimize this ruleset even with the current stack? We're not even using subchains anywhere I think, so there is a lot of repetition across our rulesets right now. That would be a low hanging fruit that we could very easily fix now (just by using ferm's @subchain), we just haven't bothered because our rulesets are still pretty tiny and the impact is negligible :)

If you're still feeling strongly about this, I think the next steps would be for you to investigate this a little further and fleet-wide (get familiar with our rulesets broadly, use e.g. cumin on classes of systems to measure their rulesets) and figure out the most pathological cases, i.e. the ones may benefit the most. Then we can discuss further (and with their maintainers as well) and choose one or two to further experiment with and measure/demonstrate the benefits. At the same time, we'll practically see how a migration would look like in terms of code and effort. Does that sound good to you?

Separately, for the Pybal/IPVS stuff, I think this could benefit to being discussed at a separate task (since it's not about iptables or firewalling) and with the involvement of people more familiar with that work (e.g. @mark, @ema, @BBlack etc.).

aborrero added a comment.Mar 1 2018, 5:27 PM
In T187994#4014123, @faidon wrote:

First, I don't think we should be thinking in terms of "using software from the 90s", at least not for something that is still as widely used and well-maintained as iptables (and to something that is as seldomly used as nftables). This is not something we should judge software with; we can talk instead in terms of amounts of bugs, maintainability, upstream response times, when was the last release, if/when it was deprecated by upstream(s) etc.

When I repeat the 90' thing is on purpose, in the sense that is has some actual consequences to iptables itself. It was designed when the linux kernel had little to no usage in datacenters, when IPv6 wasn't a thing, etc.

For code maintainability this has consequences as well: the iptables code was copy&pasted into ip6tables, and into arptables and again into ebtables. Then, every tool diverged, so we have 4 similar sourcecodes for doing mostly the same which can't be applied the same patch. All the protocol matching code (payload matching) is as well duplicated (you have a module for TCP, a module for UDP, a module for SCTP, a module for ICMP, etc..) all this code is, again, duplicated. To solve a potentially simple bug, you need a lot of code updates, and this is a burden for maintainability. Also, by design, all complexity lives in the kernel (i.e, protocol logic), therefore this leads to many kernel updates (and reboots). I described 3 of the most common issues regarding iptables being an old tool.

And these are the reasons behind nftables. It's not a fancy tool because why not. It unifies a lot of code for different families, operations and protocol/payload matching; the kernel side of the framework is very little compared to iptables; complexity is in userspace, where is much easier to apply a patch and upgrade a running software, etc.
Currently iptables is not deprecated, is still maintained, but the roadmap of the Netfilter project is nftables. iptables users are encouraged to move to nftables.

So, to sum up, I would say that from the software engineering / software project point of view, the judgment is clear: migrate :-)

I'm in a hurry right now, will reply to the other part of your comment in other moment.

MoritzMuehlenhoff added a comment.Mar 2 2018, 7:40 AM

If we had no firewall setup and would start from scratch it would be different, but I don't think the work necessary to migrate outweighs the potential benefits at this point. Our use of iptables/ferm is dead simple and "just works", we're not running into any significant problems. Same performance-wise. And even though the kernel side has lots of similar code paths, the code base is proven and rock-solid, there haven't been any notable security issues in iptables in Linux since 13 years. Iptables is also far from abandoned, the last release of the user space tools was only last month. And Linux upstream is very conservative in retaining/maintaining existing functionality anyway.

There's probably a use case for nftables in Nova which sets up it's own firewall rules (and the labcontrol hosts don't use ferm for that reason). But that would need to be implemented in OpenStack (and upstreamed to prevent rebasing with every release).

• ema added a comment.Mar 2 2018, 11:46 AM
In T187994#4014123, @faidon wrote:

Separately, for the Pybal/IPVS stuff, I think this could benefit to being discussed at a separate task (since it's not about iptables or firewalling) and with the involvement of people more familiar with that work (e.g. @mark, @ema, @BBlack etc.).

I do understand that nftables is great software, and probably the right toolset to use when writing new systems nowadays. However, switching from LVS to nftables would mean significantly altering the way our load balancers are currently working in terms of PyBal and kernel code and operational experience, to solve a problem we do not have.

The benefits of using nftables instead of LVS for load balancing, in my understanding, are in terms of reduced CPU usage (~10x IPv4 and ~6x IPv6). On our load balancers there is no sign of CPU usage being a problem at all: take for example the busiest, lvs3001, with CPU usage consistently well below 5%. While it would of course be desirable to decrease resource usage even further, it is currently not a wise way to spend our time in my view, in particular considering the bigger fishes we have to fry on that side of our infrastructure.

BBlack added a comment.Mar 2 2018, 12:31 PM

And just to put the nail in the coffin of LVS/IPVS-level concerns being raised in this ticket - if we were to look at replacing IPVS as the underlying (kernel-level) mechanism for our loadbalancers, nftables would probably not be our target. The most-promising avenue for long-term IPVS replacement that fits our feature-needs / perf / flexibility targets, and might stabilize by the time we're ready for such a transition, is the eBPF-based capabilities of XDP ( http://prototype-kernel.readthedocs.io/en/latest/networking/XDP/introduction.html ).

faidon added a comment.Mar 2 2018, 3:21 PM

To answer my own earlier question: I was looking at nftables' wiki about the supported features compared to xtables and the updates to the Linux kernel per version. Several systems (mostly WMCS) are still using trusty and Linux 3.13, which is really the first release of nftables and with multiple pretty basic features missing (e.g. REJECT, MASQUERADE etc.). Our latest and greatest right now is 4.9, and even that is apparently missing NOTRACK (added in 4.10), which is something we're using in a few places (e.g. DNSes).

So, yeah, it does look a little too bleeding edge and not there yet, at least for our velocity :/

BBlack added a comment.Mar 2 2018, 6:17 PM

https://lwn.net/Articles/747551/ has some interesting discussion on related topics, too.

aborrero added a comment.Mar 5 2018, 12:06 PM

EOF. I propose we follow up in the future.

aborrero changed the task status from Open to Stalled.Mar 5 2018, 12:07 PM
Dzahn subscribed.Oct 12 2018, 7:02 PM

bpfilter made it into 4.18 kernel and there are claims that it would "eventually replace both iptables and nftables"

aborrero added a comment.Oct 12 2018, 8:44 PM

From https://blogs.gnome.org/dcbw/2018/07/27/the-ascendance-of-nftables/ :

What about eBPF?

You might have heard that eBPF will replace everything and give everyone a unicorn. It might, if/when it gets enhancements for accountability, traceability, debuggability, auditability, and broad driver support for XDP. But nftables has been around for years and has most (all?) of these things today.
In T187994#4662164, @Dzahn wrote:

bpfilter made it into 4.18 kernel and there are claims that it would "eventually replace both iptables and nftables"

So I guess for a random value of "eventually".

BTW, iptables 1.8.0 is already in Debian (experimental), which uses the nf_tables kernel subsystem by default. If I find the time to work on it, iptables (the userspace syntax) may use nf_tables kernel backend by default for Debian Buster. https://tracker.debian.org/pkg/iptables
And that's what I said earlier, me might end using ferm->iptables->nf_tables, which doesn't make a lot of sense to me. But we can discuss further when Debian Buster is released (and we start using it).

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:48 PM
• GTirloni added a project: cloud-services-team (Kanban).Mar 21 2019, 8:17 PM
Dzahn added a comment.Jun 28 2019, 4:46 PM
In T187994#3992672, @faidon wrote:

However, iptables is being replaced by nftables.

It seems to me like nftables is still not very widely used (as also evidenced by the upstreams you mentioned not having adopted it yet) and we might be early adopters; is that your impression as well?

It seems things have changed meanwhile. Based on:

"iptables is being replaced by nftables starting with Debian Buster "

"Starting with Debian Buster, nf_tables is the default backend when using iptables, by means of the iptables-nft layer (i.e, using iptables syntax with the nf_tables kernel subsystem). This also affects ip6tables, arptables and ebtables.

You can switch back and forth between iptables-nft and iptables-legacy by means of update-alternatives (same applies to arptables and ebtables):

  1. update-alternatives --set iptables /usr/sbin/iptables-legacy
  2. update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy"

[1] https://wiki.debian.org/iptables

So with buster nftables is now the default unless we actively switch to legacy iptables.

That seems to me like we should reconsider this now that we are starting to install buster on more machines.

faidon closed this task as Declined.Jun 28 2019, 10:48 PM

I think there's a bit of a confusion. AIUI, nftables can refer to two different things:

  1. The nf_tables kernel subsystem
  2. The nftables userspace tool, which interfaces with (1)

In addition to this, there is an iptables-nft compability layer that has the same userspace interface, but that uses the nf_tables kernel subsystem (this is what what you saw in the changelog). The use of iptables-nft should be transparent, and, in fact, we are already using it on our existing buster systems.

I don't believe that there is any point for us to switch to an entirely new userspace interface at this moment. As explained above there is little benefit -- and with iptables-nft in buster, there is even less of a benefit now.

I'm marking this is as Declined because I think this task was requesting (2) above. As far as (1) is concerned this is actually Resolved (or will be in time as busters systems roll out).

jbond mentioned this in T279683: Investigate iptables replacements.Apr 29 2021, 2:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL