The current puppet configuration for releases1002 and release2002 disable docker's built in iptables rules but don't include our profile::docker::builder which provides such rules via ferm. Without proper iptables nat rules, container networking is effectively disabled.
We should either set iptables: true in profile::docker::engine::settings for releases hosts, or include the builder profile.
Including profile::docker::builder would be wrong since that also pulls in a bunch of other stuff to build the production-images that isn't needed on releases hosts. But moving that ferm rule elsewhere so it can be reused seems reasonable?
Change 670286 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] builder/docker: break out docker ferm rules into own profile
Change 670289 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] releases: include profile::docker::ferm in releases role
Change 670286 merged by Dzahn:
[operations/puppet@production] builder/docker: break out docker ferm rules into own profile
Change 670289 merged by Dzahn:
[operations/puppet@production] releases: include profile::docker::ferm in releases role
Mentioned in SAL (#wikimedia-operations) [2021-03-10T21:53:29Z] <mutante> ferm/iptables docker NAT rules applied by puppet on releases servers after breaking out fules into their own profile class (T276869)
[releases1002:~] $ sudo iptables -L | grep DOCKER DOCKER-ISOLATION all -- anywhere anywhere DOCKER all -- anywhere anywhere Chain DOCKER (1 references) Chain DOCKER-ISOLATION (1 references) [releases2002:~] $ sudo iptables -L | grep DOCKER DOCKER-ISOLATION all -- anywhere anywhere DOCKER all -- anywhere anywhere Chain DOCKER (1 references) Chain DOCKER-ISOLATION (1 references)
done. the ferm rules have been moved into profile::docker::ferm and that profile is now included in the releases role and the iptables rules exist.
Yes! Thanks so much for the fix.
I've verified that traffic is now being properly routed from the container network through the host on releases1002.
I'm seeing errors during apt-get update now for just http://security.debian.org. However I believe this may be a separate issue and doesn't necessarily block us. I'll look for a relevant task and comment there.
Err:10 http://security.debian.org buster/updates InRelease Could not connect to security.debian.org:80 (151.101.130.132), connection timed out