Page MenuHomePhabricator

Reverse DNS for k8s pods IPs
Closed, ResolvedPublic

Description

Today while debugging tegola / thanos-swift interactions I noticed that k8s pod IP ranges have no PTR records in DNS, making debugging a little harder IMHO.

This task is thus to propose having PTRs synthesized for k8s pods IP ranges (i.e. even pointing to the same name I think would be helpful). cc T270071: SVC DNS zonefiles and source of truth as a related/adjacent task

Event Timeline

JMeybohm added subscribers: Volans, BBlack, JMeybohm.

From IRC discussion today:

  • we do have already some for loops in the dns that generate kubernetes-pod-10-64-x-y.eqiad.wmnet records (grep for 'range' in dns repo)
  • the ranges are probably no longer correct
  • the ranges are missing aux, dse and ml clusters
  • the name is not ideal as it does not refer to a specific cluster, suggestion is to use 10-16-1-234.pods.<CLUSTER-GROUP>.<DC>.wmnet
CDanis subscribed.

I think in light of T370304 we need to prioritize this -- and a proper version, not just creating static placeholder records.

The good news is, we already have the data available.


Our CoreDNS in k8s is already configured to serve all the PTRs it knows how to serve (it's configured as being authoritative for the in-addr.arpa zone).

One wrinkle is that technically CoreDNS doesn't serve PTRs for all pod IPs -- only IPs that are listed in Endpoints on k8s. In effect this distinction works out to "pod IPs of pods that accept incoming connections".

I took a random sample of 200 pod IPs across all namespaces in eqiad wikikube. I excluded the calico-node pods because they're hostNetwork, and have the endpoint of the node IP.
Then, after using my favorite teleprinter-native data interchange format, I reverse-resolved all 200 in a wmfdebug pod I created for this purpose.
(Full method at P67405 if you're curious.)

The results (in a TSV of IP, pod name, and PTR record if any) are in paste P67404.

Only 16/200 failed to reverse-resolve. And in this sample, those pods were things like k8s cluster infrastructure, some Flink controllers, eventstreams, plus my debug pod.


So I think we could change our kube-dns Service to be of type NodePort to (sorta) expose this to the rest of production.
Maybe also Puppetize a DNAT rule to redirect incoming packets to the node's port 53 to that NodePort?
That's actually Good Enough™ for debugging things like "what's hammering my database master" -- CoreDNS does recurse if it needs to, so you could set a few k8s nodes as the nameservers for your database machine.

In terms of a not-gross way to make it broadly usable within production, though, there's ... a lot more to consider. But the NodePort step seems like a necessary prereq for any method we might choose.

For the record: The CoreDNS Pods are, like all other Pods, reachable by their Pod IP. So going through a debug Pod to resolve the PTRs is not really necessary.
Regarding the IPs of Pods not listed in Endpoints: These could be added to CoreDNS via the https://github.com/coredns/kubepods plugin as well.

Changing the coredns kubernetes service to NodePort seems like an easy thing to do, as we (un)fortunately allow NodePorts in the rage 1-65535 for historical reasons. That would take care of the DNAT as well (at that's what a NodePort is for).

While I do agree with this making lives easier, I'm generally not a fan of making (some) k8s nodes special in any way that prevents us from treating them like we do now (e.g depool/shutdown whenever we want without considering things outside k8s). We could potentially use the kubernetes apiservers for this as they are special anyways and we usually make sure not to turn them down all at once. But when a full k8s cluster is depooled, that assumption no longer holds.

It might be quite naive (and does not solve my concern from above) but could we have a subnet delegation or a forwarding for the K8s Pod-IP ranges in PowerDNS to make this transparent for the rest of the infrastructure?

For the record: The CoreDNS Pods are, like all other Pods, reachable by their Pod IP. So going through a debug Pod to resolve the PTRs is not really necessary.

🤦 good point lol

Regarding the IPs of Pods not listed in Endpoints: These could be added to CoreDNS via the https://github.com/coredns/kubepods plugin as well.

I came across that too, but no commits in the last 2.5 years made me go hmmm.

Changing the coredns kubernetes service to NodePort seems like an easy thing to do, as we (un)fortunately allow NodePorts in the rage 1-65535 for historical reasons. That would take care of the DNAT as well (at that's what a NodePort is for).

While I do agree with this making lives easier, I'm generally not a fan of making (some) k8s nodes special in any way that prevents us from treating them like we do now (e.g depool/shutdown whenever we want without considering things outside k8s). We could potentially use the kubernetes apiservers for this as they are special anyways and we usually make sure not to turn them down all at once. But when a full k8s cluster is depooled, that assumption no longer holds.

I think it's fine for this to be unreachable while a whole cluster is down? CoreDNS and the k8s apiservers have a lot of other shared fate anyway. And in the case that a k8s cluster is down, there shouldn't be much need to reverse-resolve pod IPs.

Treating the apiservers as special for this use case was my instinct too.

It might be quite naive (and does not solve my concern from above) but could we have a subnet delegation or a forwarding for the K8s Pod-IP ranges in PowerDNS to make this transparent for the rest of the infrastructure?

Yeah, having pdns_recursor forward the zone was basically what I was thinking. Naive but easy and effective.

We might still need to solve the problem of pdns healthchecking the nodes we're giving it the IPs of. I'm not finding much concrete in the pdns_recursor docs about balancing amongst servers that can go down, but there are a few things that make it seem like it might just work -- forward-zones can accept multiple IP addresses for the same zone, and there's also the outgoing.server_down_max_fails / server-down-max-fails config knob.

I guess another option would be using dnsdist as the way to balance amongst the nodes, and then point pdns_recursor to that -- dnsdist has good docs and configuration for its healthchecking of backends. Maybe @Vgutierrez or @ssingh have opinions?

As it turns out this was discussed at the k8s SIG but the bug didn't get updated. Meeting notes

In summary Alex is looking at configuring Calico's IPAM to keep a set of coredns pods on static IP addresses, which we can then use outside the cluster. And then we'll just use standard zone delegation from gdnsd authservers.

It might be quite naive (and does not solve my concern from above) but could we have a subnet delegation or a forwarding for the K8s Pod-IP ranges in PowerDNS to make this transparent for the rest of the infrastructure?

Yeah, having pdns_recursor forward the zone was basically what I was thinking. Naive but easy and effective.

I'm not a massive fan of doing custom re-direction on recursive dns servers unless there is no alternative. I don't see a reason here why we couldn't have standard zone delegation from the parent in-addr.arpa and ip6.arpa zones to coredns? For example something like this in the 10.in-addr.arpa zone template in the dns repo should delegate the reverses for the 10.67.80.0/24 range:

80.67    IN  NS  <coredns_ip1>
80.67    IN  NS  <coredns_ip2>

We'd need to repeat that for every /24 assigned to cover all the pod ranges assigned. IPv6 is a little easier - though ugly as ever - as there is just a /64 assigned per site to each group. So something like the following in 0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa would I think cover the eqiad range 2620:0:860:cabe::/64:

e.a.0.8.5.e.7.5.1.b.5.a.8.1.b.2.e.b.a.c  IN  NS  <coredns_ip1>
e.a.0.8.5.e.7.5.1.b.5.a.8.1.b.2.e.b.a.c  IN  NS  <coredns_ip2>

In terms of failures / retries if a particular coredns IP is unresponsive we can probably then rely on the standard dns timeout/retry mechanisms (with caching)?

I may be completely missing something important here however, but in general if we can do it via zone delegation rather than custom config on the recursors it's cleaner.

In summary Alex is looking at configuring Calico's IPAM to keep a set of coredns pods on static IP addresses, which we can then use outside the cluster. And then we'll just use standard zone delegation from gdnsd authservers.

I probably should have read this comment first :D

As best I know here's the current state of this task:

A couple weeks ago we discussed on -k8s-sig IRC that the "some coredns pod IPs get assigned statically" mechanism proposed by @akosiaris is unsatisfactory because it means we can't use Deployments and would have to create (and manage?) bare Pod objects ourselves (or I guess create and manage a group of single-replica ReplicaSets). @JMeybohm commented that this seemed even worse than making the k8s control plane apiservers special in some way, and I agree.

I don't think any other progress has been made or other ideas have been had since?

So, I think this leaves us with two easy and viable options.

  1. Also run coredns as a daemonset limited to control plane nodes. Expose them there as NodePort 53. If we wanted, we could also add a pod selector for this DS to the existing coredns Service, and then scale down the existing coredns Deployment by 3. DNS-delegate PTR zones to the control plane servers.
  2. For this use case only, experiment with Calico advertising coredns's service IP, while also using externalTrafficPolicy: Local so we avoid both SNAT and any extra hops through nodes in the cluster. DNS-delegate PTR zones to the ClusterIP.
    • @akosiaris raised a concern that Calico open core lacks support for both BGP graceful restart as well as using BFD. This means that any node reboots/crashes will blackhole traffic until the BGP timers expire. However ... I think that's fine? This is internal-only DNS for debugging, it's short-term cacheable, all clients have reasonable default retry behavior, and we set hold-time to 30 seconds on our cr<>k8s BGP sessions anyway.

I have a preference for the latter but I guess I could be talked into the former.

But I really do want us to fix this, it's only a matter of time before we once again turn the appservers into a death ray firing at some non-k8s part of production. And soon this will be blocking I/F hackathon work on T372943.

As discussed at the k8s SIG today:

  • There were some doubts about if Calico can advertise just a subset of Service ClusterIP range, or if it was all-or-nothing
    • I think this might have been something that used to be true but changed -- docs for 3.23.3 (our version in prod) mention a deprecated CALICO_ADVERTISE_CLUSTER_IPS environment variable that sounds like worked this way
    • the BGPConfiguration object in 3.23.3 has a serviceClusterIPs field that's a list of CIDRs, and there's no mention on the reference or tutorial page that this can't be a /32 or a /128
  • There was uncertainty about whether advertising the ClusterIP might wind up interfering with kube-proxy within the cluster
    • Yeah, this one is a bit harder to be confident about, aside from the Calico documentation and the k8s documentation for this feature don't have obvious warnings.

Right after the meeting I stumbled across one other promising option: using the externalIPs field of Services.

  • They are to be the predecessor to the whole LoadBalancer abstraction, as they're a field set on the Service, but ofc the addresses must be provisioned by the cluster operator
  • That's fine, let's just pick some unused ranges in our various per-cluster private IP space and call them our outside-k8s service IPs
  • Calico 3.23.3 supports it and the reference example shows advertising a /32 externalIP

I think this answers any doubt about kube-proxy interactions or unintended advertisements?

Change #1075311 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] coredns: add support for Service externalIPs

https://gerrit.wikimedia.org/r/1075311

Janis agrees with me that this use case seems like a great excuse to experiment with externalIPs.

So I suggest the following:

  • @ayounsi and I will pick six small unused ranges in our current rfc1918 wikikube allocations (v4+v6 for each of eqiad, codfw, staging). I don't think anything larger than a v4 /24 makes sense; smaller would be fine.
    • That IP space will be managed by hand in Netbox. Pick one IP address in each for CoreDNS.
  • Redeploy coredns in staging with externalIPs enabled and set appropriately
  • Patch the Calico default BGPConfiguration in staging to include (for example):
serviceExternalIPs:
- cidr: 10.64.77.53/32
- cidr: 2620:0:860:babd::53/128

I think we'll probably also want to add those ranges to prefixAdvertisements with the k8s community?

  • If all looks good, repeat in eqiad and then codfw

Change #1075918 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] calico: add BGP communities to serviceExternalIPs

https://gerrit.wikimedia.org/r/1075918

Change #1076768 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] WIP

https://gerrit.wikimedia.org/r/1076768

Change #1076768 merged by CDanis:

[operations/puppet@production] experiment w/ externalIPs on staging-codfw

https://gerrit.wikimedia.org/r/1076768

Change #1076837 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] Revert "experiment w/ externalIPs on staging-codfw"

https://gerrit.wikimedia.org/r/1076837

Change #1076837 merged by CDanis:

[operations/puppet@production] Revert "experiment w/ externalIPs on staging-codfw"

https://gerrit.wikimedia.org/r/1076837

I looked into how we might generate the required NS records in the appropriate zones to delegate ranges used by k8s to the appropriate control-plane nodes for each cluster:

https://github.com/topranks/random_wmf/tree/main/hackathon_k8s_dns

I will try to speak to interested stake-holders to get input on how it might be productionized.

Change #1077043 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] CoreDNS chart changes to serve outside the cluster

https://gerrit.wikimedia.org/r/1077043

Change #1075311 abandoned by CDanis:

[operations/deployment-charts@master] coredns: add support for Service externalIPs

Reason:

manual testing unpromising

https://gerrit.wikimedia.org/r/1075311

Change #1077090 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] coredns: improve debuggability

https://gerrit.wikimedia.org/r/1077090

Change #1077043 abandoned by CDanis:

[operations/deployment-charts@master] CoreDNS chart changes to serve outside the cluster

Reason:

K.I.S.S.

https://gerrit.wikimedia.org/r/1077043

Change #1077376 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] Bug: T344171

https://gerrit.wikimedia.org/r/1077376

Change #1077376 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] coredns: support NodePort & bump chart

https://gerrit.wikimedia.org/r/1077376

Change #1077090 merged by jenkins-bot:

[operations/deployment-charts@master] coredns: improve debuggability

https://gerrit.wikimedia.org/r/1077090

Change #1075918 abandoned by CDanis:

[operations/deployment-charts@master] calico: add BGP communities to serviceExternalIPs

Reason:


https://gerrit.wikimedia.org/r/1075918

Change #1077376 merged by jenkins-bot:

[operations/deployment-charts@master] coredns: support NodePort & bump chart

https://gerrit.wikimedia.org/r/1077376

Change #1077418 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] coredns: enable nodePort 53 everywhere

https://gerrit.wikimedia.org/r/1077418

Change #1077418 merged by jenkins-bot:

[operations/deployment-charts@master] coredns: enable nodePort 53 everywhere

https://gerrit.wikimedia.org/r/1077418

OK, one weird issue I've found which is confounding but not fatal: the NodePort isn't working on v6.

Capturing on 'eno1'
    1 0.000000000 2620:0:861:102:10:64:16:93 → 2620:0:860:104:10:192:48:32 DNS 128 Standard query 0x4fb3 PTR 59.75.192.10.in-addr.arpa OPT
    2 0.997566221  10.64.16.93 → 10.192.48.32 DNS 108 Standard query 0x4fb3 PTR 59.75.192.10.in-addr.arpa OPT
    3 1.029053768 10.192.48.32 → 10.64.16.93  DNS 223 Standard query response 0x4fb3 PTR 59.75.192.10.in-addr.arpa PTR 10-192-75-59.chromium-render-production-tls-service.proton.svc.cluster.local OPT

To make progress here while we work on automating the sub-zone delegation I have manually delegated the required zones covering the k8s pod IP space to the appropriate control-plane hosts (patch).

There is an issue, however, which I've not quite got to the bottom of. Responses from gdnsd seem good:

cmooney@dns1004:~$ dig +nsid NS 75.192.10.in-addr.arpa @ns0.wikimedia.org 

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> +nsid NS 75.192.10.in-addr.arpa @ns0.wikimedia.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13936
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 113ca40ec8de2c762a1bd391beb647cb (good)
; NSID: 64 6e 73 31 30 30 34 2d 61 75 74 68 ("dns1004-auth")
;; QUESTION SECTION:
;75.192.10.in-addr.arpa.		IN	NS

;; AUTHORITY SECTION:
75.192.10.in-addr.arpa.	3600	IN	NS	kubestagemaster2003.codfw.wmnet.
75.192.10.in-addr.arpa.	3600	IN	NS	kubestagemaster2004.codfw.wmnet.
75.192.10.in-addr.arpa.	3600	IN	NS	kubestagemaster2005.codfw.wmnet.

;; Query time: 0 msec
;; SERVER: 208.80.154.238#53(ns0.wikimedia.org) (UDP)
;; WHEN: Thu Oct 03 17:18:27 UTC 2024
;; MSG SIZE  rcvd: 200

But query to our resolver IP (pdns-recursor) gets a SERVFAIL response:

cmooney@dns1004:~$ dig +nsid NS 75.192.10.in-addr.arpa @10.3.0.1 

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> +nsid NS 75.192.10.in-addr.arpa @10.3.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 37272
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; NSID: 64 6e 73 31 30 30 34 ("dns1004")
;; QUESTION SECTION:
;75.192.10.in-addr.arpa.		IN	NS

;; Query time: 3 msec
;; SERVER: 10.3.0.1#53(10.3.0.1) (UDP)
;; WHEN: Thu Oct 03 17:18:33 UTC 2024
;; MSG SIZE  rcvd: 62

I thought it was probably due to something coredns was returning, but having checked a few times I don't believe that pdns-recursor is making any attempt to query the kubectl servers gdnsd returns.

Looks like the pdns-recursor doesn't like the answer from gdnsd, but I can't tell why. The WMCS pdns-recursor also gave me a similar result fwiw.

Since gdnsd is fine and pdns-recursor is not, could it be because of this?

https://doc.powerdns.com/recursor/settings.html#setting-dont-query

Default: 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fc00::/7, fe80::/10, 0.0.0.0/8, 192.0.0.0/24, 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 240.0.0.0/4, ::/96, ::ffff:0:0/96, 100::/64, 2001:db8::/32
Queries for names in forward zones and to addresses as configured in any of the settings forward-zones, forward-zones-file or forward-zones-recurse are performed regardless of these limitations. However, if NS records are learned from forward-zones and the IP addresses of the nameservers learned in that way are included in dont-query, lookups relying on these nameservers will fail with SERVFAIL.

Which means we will need to explicitly set dont-query to exclude the private addresses.

If we want to confirm the above, we can depool a DNS host, disable Puppet, manually edit recursor.conf, restart it, test it and then revert. It should not be a problem on a depooled host and losing the cache is not a big deal.

@ssingh yes that I think is probably it!

Totally makes sense for the recursor to have that rule. People often mess up and put those unreachable IPs in DNS data, and they could even be used maliciously to maybe cause resource exhaustion (esp to 127.0.0.1 etc).

If the kubectl servers were running the service properly on IPv6 this would not be an issue, but unfortunately they are only running on IPv4. So the 10.x.x.x address of say aux-k8s-ctrl1001.eqiad.wmnet. looks suspect to pdns and it doesn't make the query.

The question is what to do? Removing 10.0.0.0/8 from the list would hopefully fix it, but not sure if there are any caveats or worries there. Or otherwise we might need to try to allocate public IPs for this (or get coredns working on v6).

FWIW I tested the delegation of one of the v6 ranges with "dig +trace" and that does also suggest gdnsd is ok and the delegation is working - P69460

Mentioned in SAL (#wikimedia-operations) [2024-10-03T18:28:37Z] <sukhe> depool dns1005 for all services for testing T344171

Are the kubectls IP in some particular subnet? If so, we can exclude them from above, such as !10.64.0.0/12 or something -- you get the idea! Or get v6 working for coredns, whichever is easier.

@ssingh you were correct!!! awesome <3

We depooled dns1005 and then modified /etc/powerdns/recursor.conf, adding a dont-query statement with all the default networks apart from 10.0.0.0/8. With that done queries to the local service for the k8s IPs work!!

cmooney@dns1005:/etc/powerdns$ dig +nsid -x 10.192.75.59 @10.3.0.1 

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> +nsid -x 10.192.75.59 @10.3.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50676
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; NSID: 64 6e 73 31 30 30 35 ("dns1005")
;; QUESTION SECTION:
;59.75.192.10.in-addr.arpa.	IN	PTR

;; ANSWER SECTION:
59.75.192.10.in-addr.arpa. 5	IN	PTR	10-192-75-59.chromium-render-production-tls-service.proton.svc.cluster.local.

;; Query time: 31 msec
;; SERVER: 10.3.0.1#53(10.3.0.1) (UDP)
;; WHEN: Thu Oct 03 18:35:35 UTC 2024
;; MSG SIZE  rcvd: 155

Are the kubectls IP in some particular subnet? If so, we can exclude them from above, such as !10.64.0.0/12 or something -- you get the idea! Or get v6 working for coredns, whichever is easier.

They're not on any particular subnet unfortunately, just on the wmf private vlan in whatever row/rack they are installed. We only have k8s in eqiad and codfw, so I guess we could include the private ranges we have allocated for those, 10.64.0.0/16 and 10.192.0.0/18 are the smallest networks we could simply limit it to looking at Netbox (we've so many /24s now with the per-rack subnets). But definitely might be good to do that if we take that approach, we should only loosen the restriction by the minimal necessary.

The great news is the delegation on gdnsd is working correctly with the manual entries, and we've some decent ideas on how to automate their creation so they don't stay manual long-term.

Mentioned in SAL (#wikimedia-operations) [2024-10-03T18:51:26Z] <cmooney@cumin1002> conftool action : set/pooled=yes; selector: name=dns1005.wikimedia.org [reason: testing T344171]

Continuing on the above: I was wondering if we should even worry about setting dont-query to !10.0.0.0/8 or negating any other private addresses. Because unlike gdnsd, we set an ACL for pdns-recursor, which means that clients outside of:

# If set, only allow these comma separated netmasks to recurse
allow-from=208.80.152.0/22, 2620:0:860::/46, 198.35.26.0/23, 185.71.138.0/24, 2001:67c:930::/48, 2a02:ec80::/32, 2001:df2:e500::/48, 103.102.166.0/24, 185.15.58.0/24, 185.15.59.0/24, 195.200.68.0/24, 10.0.0.0/8, 127.0.0.0/8, ::1/128

... will not be able to recurse anyway and this supersedes dont-query, which kicks in later as the client's query will be refused based on the ACL. This ensures sufficient protection (we depend on the ACL for other pdns-recursor restrictions as well) and against being worried about setting !10.0.0.0/8 but at least we can set something smaller like !10.64.0.0/16 and !10.192.0.0/18 without much worry IMO.

Continuing on the above: I was wondering if we should even worry about setting dont-query to !10.0.0.0/8 or negating any other private addresses. Because unlike gdnsd, we set an ACL for pdns-recursor

Indeed yes, we are not operating an open-resolver to the world, and it's only available on a private IP (10.3.0.1). So we won't have any malicious user sending crafted queries about deliberately bad domains in an attempt at resource exhaustion or denial-of-service.

Which I think largely means we are ok here. Of course some elements on our network may try to request internet domains which are misconfigured, with listed name servers on those ranges, causing pdns-recursor to make futile attempts to connect to them, or make dns requests to some of our own hosts in 10.x which happen to be the same as some misconfigured domain's NS records.

But I agree I think that is a small risk, thanks for pointing that out!

OK, one weird issue I've found which is confounding but not fatal: the NodePort isn't working on v6.

Capturing on 'eno1'
    1 0.000000000 2620:0:861:102:10:64:16:93 → 2620:0:860:104:10:192:48:32 DNS 128 Standard query 0x4fb3 PTR 59.75.192.10.in-addr.arpa OPT
    2 0.997566221  10.64.16.93 → 10.192.48.32 DNS 108 Standard query 0x4fb3 PTR 59.75.192.10.in-addr.arpa OPT
    3 1.029053768 10.192.48.32 → 10.64.16.93  DNS 223 Standard query response 0x4fb3 PTR 59.75.192.10.in-addr.arpa PTR 10-192-75-59.chromium-render-production-tls-service.proton.svc.cluster.local OPT

This is because the kube-dns service is configured with ipFamilyPolicy: SingleStack (probably not on purpose but because it is the default). Switching to PreferDualStack should add an IPv6 IP to the service and bind the NodePort. Keep in mind that k8s clusters other than wikikube don't have dual stack networking enabled right now (ipv6dualstack in hieradata/common/kubernetes.yaml).

Change #1078716 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] recursor: Use NSes in eqiad+codfw private IP space

https://gerrit.wikimedia.org/r/1078716

Change #1078719 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] ferm: allow DNS traffic against k8s control planes

https://gerrit.wikimedia.org/r/1078719

Mentioned in SAL (#wikimedia-operations) [2024-10-08T18:40:54Z] <cdanis> 💙cdanis@cumin1002.eqiad.wmnet ~ 🕝☕ sudo cumin A:dnsbox 'disable-puppet "cdanis rolling out T344171 Ie7d509"'

Change #1078716 merged by CDanis:

[operations/puppet@production] recursor: Use NSes in eqiad+codfw private IP space

https://gerrit.wikimedia.org/r/1078716

Mentioned in SAL (#wikimedia-operations) [2024-10-08T18:43:13Z] <cdanis> 💔cdanis@cumin1002.eqiad.wmnet ~ 🕝☕ sudo cumin -b1 -s120 A:dnsbox 'run-puppet-agent --enable "cdanis rolling out T344171 Ie7d509"'

Works in prod now:

1; <<>> DiG 9.16.50-Debian <<>> -x 10.192.75.58 +nsid
2;; global options: +cmd
3;; Got answer:
4;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59584
5;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
6
7;; OPT PSEUDOSECTION:
8; EDNS: version: 0, flags:; udp: 512
9; NSID: 64 6e 73 31 30 30 36 ("dns1006")
10;; QUESTION SECTION:
11;58.75.192.10.in-addr.arpa. IN PTR
12
13;; ANSWER SECTION:
1458.75.192.10.in-addr.arpa. 5 IN PTR 10-192-75-58.termbox-test.termbox.svc.cluster.local.
15
16;; Query time: 36 msec
17;; SERVER: 10.3.0.1#53(10.3.0.1)
18;; WHEN: Tue Oct 08 21:19:32 UTC 2024
19;; MSG SIZE rcvd: 130

Remaining work to do:

Change #1078719 abandoned by CDanis:

[operations/puppet@production] ferm: allow DNS traffic against k8s control planes

https://gerrit.wikimedia.org/r/1078719

CDanis claimed this task.
root@db1169:~# ss -tr | grep :mysql
ESTAB 0      945     db1169.eqiad.wmnet:mysql   10-67-163-233.mediawiki.mw-api-ext.svc.cluster.local:57846            
ESTAB 0      0       db1169.eqiad.wmnet:mysql       10-67-163-180.mediawiki.mw-web.svc.cluster.local:54236            
ESTAB 0      0       db1169.eqiad.wmnet:mysql        10-67-172-94.mediawiki.mw-web.svc.cluster.local:40890            
ESTAB 0      0       db1169.eqiad.wmnet:mysql   10-67-163-151.mediawiki.mw-api-ext.svc.cluster.local:42186            
ESTAB 0      0       db1169.eqiad.wmnet:mysql   10-67-133-187.mediawiki.mw-api-int.svc.cluster.local:57268            
ESTAB 0      674     db1169.eqiad.wmnet:mysql   10-67-190-118.mediawiki.mw-api-ext.svc.cluster.local:42074            
ESTAB 0      11      db1169.eqiad.wmnet:mysql    10-67-158-92.mediawiki.mw-api-ext.svc.cluster.local:33186            
ESTAB 0      753     db1169.eqiad.wmnet:mysql       10-67-172-220.mediawiki.mw-web.svc.cluster.local:60126            
ESTAB 0      0       db1169.eqiad.wmnet:mysql       10-67-189-105.mediawiki.mw-web.svc.cluster.local:32812            
ESTAB 0      673     db1169.eqiad.wmnet:mysql        10-67-167-12.mediawiki.mw-web.svc.cluster.local:35374