Blocking DNS to the pod entirely would cause a lot of headaches,

Actually, this is worth looking more closely at. We need DNS currently for resolving "main-opentelemetry-collector.opentelemetry-collector.svc.cluster.local" in the tracing config, but in theory we could just inject a static IP. Same goes for the orchestrator address for T427863. My naive instinct is that a bunch of other infra-level stuff would break if we cut off the pod's access to DNS, but maybe that's not true?

I'm definitely happy to just inject the IP for the OTel access point and similar, but is it actually a round-robin/etc. for which we're relying on DNS giving different answers from time to time? Or is that no longer used anywhere?

Those "svc.cluster.local" addresses are virtual IPs that don't change, although they are cluster-dependent -- for example, the orchestrator service's cluster IP is always 10.64.72.154 in eqiad and 10.192.72.115 in codfw. (via "kubectl get service"). Traffic to those IPs gets load-balanced to different pods as they come and go.

So that's why the static IP approach might be viable for those two use cases, and it looks like we could even leave those hostnames as-is and use the pod's .spec.hostAliases to insert them in /etc/hosts. But I don't know if we're relying on a more active DNS for anything else.

We use cluster wide policy (GlobalNetworkPolicy) deny-all to block everything but DNS (to the kube-dns service) which is configured to forward to our DNS infra (whatever is configured in the worker nodes /etc/resolv.conf really).

Blocking DNS entirely and relying on hostAliases has the downside of being very snowflaky which might bite us when we've forgotten about it and try to debug something strange. Since ClusterIPs can change (up until now they actually did change regularly during Kubernetes upgrades), this needs additional awareness.

An alternative could be to spin up a separate CoreDNS deployment that only forwards .wmnet via /etc/resolv.conf and configure the wikifunctions to use that instead of the default DNS service. Internal lookups would still work like they used to but external won't. This has the obvious downside of having to run a second CoreDNS deployment (not that big of a deal) and still having a special DNS config for wikifunctions (which we'd need either way).

Regardless of what we choose to do, we should move the DNS allow rules from deny-all to a new rule which does not apply to the wikifunctions namespace.

# coredns
rzl@kubestage2003:~$ dig +noall +answer @10.192.76.3 cumin2003.codfw.wmnet
cumin2003.codfw.wmnet.	30	IN	A	10.192.15.6
rzl@kubestage2003:~$ dig +noall +answer @10.192.76.3 google.com
google.com.		30	IN	A	142.250.113.113
google.com.		30	IN	A	142.250.113.139
google.com.		30	IN	A	142.250.113.138
google.com.		30	IN	A	142.250.113.101
google.com.		30	IN	A	142.250.113.102
google.com.		30	IN	A	142.250.113.100


# coredns-internalonly
rzl@kubestage2003:~$ dig +noall +answer @10.192.76.4 cumin2003.codfw.wmnet
cumin2003.codfw.wmnet.	30	IN	A	10.192.15.6
rzl@kubestage2003:~$ dig +noall +answer @10.192.76.4 google.com
rzl@kubestage2003:~$

Works!

Wikifunctions is now using kube-dns-internalonly.

I belatedly realized I had to split up the GlobalNetworkPolicy patch into two: https://gerrit.wikimedia.org/r/1305799, at PS3, both enabled access to the new DNS and restricted access to the old one. That wasn't going to work: instead we needed to enable access to the new one (that's the same patch, at PS4), then switch wikifunctions to use it, then restrict access to the old one (that's now https://gerrit.wikimedia.org/r/1309293). Thanks @Scott_French for jumping in for a speedy review of that change.

After the switch, the basic checks in check-wf-services.py are happy, and nothing seems wrong in metrics or logs. We can let this bake for a few days, then apply the restricted GNP to actually get the security benefits we came for, then call this resolved.

As a followup, serviceops may be interested in making kube-dns-internalonly the default, since most services don't actually need external DNS. We can explore that in a separate ticket.