Page MenuHomePhabricator

10.67.28.73 reverse DNS showing 2(SERVFAIL)
Closed, ResolvedPublic

Description

Whilst troubleshooting a database issue, I found the following IP (being active) on show processlist: 10.67.28.73
When checking which host this was, in order to ssh and debug locally I ran into:

marostegui@cumin1003:~$ host 10.67.28.73
Host 73.28.67.10.in-addr.arpa not found: 2(SERVFAIL)

The following IRC conversation happened:

[11:26:20]  <marostegui> elukey is this normal? 
[11:26:21]  <marostegui> [09:25:57] marostegui@cumin1003:~$ host 10.67.28.73
[11:26:21]  <marostegui> Host 73.28.67.10.in-addr.arpa not found: 2(SERVFAIL)
[11:27:55]  <elukey> marostegui: lovely, never seen it before. Does it happen only with that IP, or more?
[11:28:21]  <marostegui> elukey: I was debugging something and found that it, I just tried with another host and it does look fine
[11:31:13]  <elukey> marostegui: but how did you come up with the IP? Is it appearing in some log etc..? Seems a k8s one in theory, so I am wondering if IPs belonging to dead pods may end up with servfail and not nxdomain for $reason
[11:31:28]  <marostegui> elukey: yeah, it was from a show processlist in a database
[11:31:36]  <marostegui> And I wanted to check which host it was
[11:31:38]  <marostegui> To ssh there etc
[11:32:39]  <elukey> I tried other stuff like 10.67.28.71 etc.. and they end up in servfail, so I am almost sure the IP you found was related to a pod that got killed
[11:33:33]  <marostegui> elukey: but that IP is actively doing things eh
[11:35:45]  <elukey> marostegui: oh ok I didn't get this bit before
[11:36:15]  <elukey> mmmm maybe it is related to coredns on the target k8s cluster
[11:37:00]  <elukey> netbox suggests that it is a dse k8s pod https://netbox.wikimedia.org/ipam/prefixes/538/
[11:37:16]  <marostegui> elukey: The thing is a bit tricky, because there's an IP doing stuff but I cannot access it as it doesn't resolve a name
[11:37:43]  <elukey> marostegui: yeah I know, lemme try to find more info about the IP. 
[11:37:54]  <marostegui> elukey: <3
[11:43:52]  <elukey> wikidatawiki-sql-xml-wikidatawiki-dump-remaining-full-ev2anki running on dse-k8s-worker1014.eqiad.wmnet

Event Timeline

Thanks @Marostegui for the task. There are a few components here that need to work.

Firstly our authdns servers are responsible for the entire 10.0.0.0/8 range (10.in-addr.arpa). However as this is a Kubernetes IP the assignment of it is handled by the K8s control plane nodes for the given cluster, so we have the 10.67.28.0/24 reverse range delegated to those from our auth servers:

cmooney@cumin1003:~$ dig +noall +authority NS 28.67.10.in-addr.arpa. @ns0.wikimedia.org 
28.67.10.in-addr.arpa.	3600	IN	NS	dse-k8s-ctrl1001.eqiad.wmnet.
28.67.10.in-addr.arpa.	3600	IN	NS	dse-k8s-ctrl1002.eqiad.wmnet.

When I check another random IP in that range these servers do indeed return the hostname, which tells us the above delegation is correct and working:

cmooney@cumin1003:~$ host 10.67.28.1
1.28.67.10.in-addr.arpa domain name pointer 10-67-28-1.postgresql-growthbook-next-r.growthbook-next.svc.cluster.local.
1.28.67.10.in-addr.arpa domain name pointer 10-67-28-1.postgresql-growthbook-next-ro.growthbook-next.svc.cluster.local.

However our recursive DNS is returning SERVFAIL for the particular IP you were checking:

cmooney@cumin1003:~$ dig -x 10.67.28.73 @10.3.0.1

; <<>> DiG 9.18.47-1~deb12u1-Debian <<>> -x 10.67.28.73 @10.3.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 51201

If we ask the question directly to the dse k8s control plane nodes they respond without an error / SERVFAIL. But there is no 'answer' section meaning they have no reverse entry set up for that IP:

cmooney@cumin1003:~$ dig -4 +nsid -x 10.67.28.73 @dse-k8s-ctrl1001.eqiad.wmnet. 

; <<>> DiG 9.18.47-1~deb12u1-Debian <<>> -4 +nsid -x 10.67.28.73 @dse-k8s-ctrl1001.eqiad.wmnet.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48602
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; NSID: 63 6f 72 65 64 6e 73 2d 35 63 35 39 64 63 63 36 39 34 2d 62 66 6c 64 74 ("coredns-5c59dcc694-bfldt")
;; QUESTION SECTION:
;73.28.67.10.in-addr.arpa.	IN	PTR

;; Query time: 8 msec
;; SERVER: 10.64.0.228#53(dse-k8s-ctrl1001.eqiad.wmnet.) (UDP)
;; WHEN: Tue Jun 09 10:53:08 UTC 2026
;; MSG SIZE  rcvd: 81

The SERVFAIL our resolver returns in this case is because the K8S control plane servers do not return a valid SOA for the delegated /24. But we can probably ignore that, if they actually had a record for the particular IP we'd not see that.

Fix

Unfortunately after all that I don't have a fix. We need to work out why dse-k8s-ctrl1001 is able to return a reverse record for 10.67.28.1, but not for 10.67.28.73. I'm not sure if that's something to do with the way the particular POD or app is deployed on Kubernetes.

Seems the IP is in use for mediawiki-dumps-legacy:

cmooney@dse-k8s-ctrl1001:~$ sudo kubectl get pods -o wide --all-namespaces | grep 10.67.28.73
mediawiki-dumps-legacy                      wikidatawiki-sql-xml-wikidatawiki-dump-remaining-full-ev2anki     4/4     Running                      0                  2d15h   10.67.28.73    dse-k8s-worker1014.eqiad.wmnet      <none>           <none>

It doesn't seem to have a service endpoint registered though, which I think is needed before CoreDNS will publish any records for it:

cmooney@dse-k8s-ctrl1001:~$ sudo kubectl get endpointslices -A | grep 10.67.28.73
cmooney@dse-k8s-ctrl1001:~$

@CDanis not sure if you have any thoughts here? I think because this is a job and not a service endpoint there is no DNS created. And from a bit of brief reading it doesn't seem trivial to remedy.

It's a shame as the IP was found in troubleshooting, so it'd be good to be able to narrow it down. But otherwise I think there isn't much else we can do here.

It doesn't seem to have a service endpoint registered though, which I think is needed before CoreDNS will publish any records for it:

cmooney@dse-k8s-ctrl1001:~$ sudo kubectl get endpointslices -A | grep 10.67.28.73
cmooney@dse-k8s-ctrl1001:~$

This is exactly right, and I think is still a hard limitation of the CoreDNS default kubernetes plugin:

PTR Records
This plugin creates PTR records for every Pod selected by a Service. If a given Pod is selected by more than one Service a separate PTR record will exist for each Service selecting it.

It might be possible to work around this by creating headless services for these jobs that access the databases -- definitely worth trying.

Another option to look at would be the official-unofficial https://github.com/coredns/kubepods plugin.

I'll take this to the Kubernetes SIG.

Thanks @CDanis

Regarding the SERVFAIL itself I think the reason that happens is that coredns is configured to "fallthrough" if it gets a PTR query for a record it doesn't have.

kubernetes cluster.local in-addr.arpa ip6.arpa {
    pods insecure
    fallthrough in-addr.arpa ip6.arpa
}

Which probably is what it should do - if anything is using it as a resolver - so that it will fetch PTR records from upstream for those it's not responsible for.

I suspect the SERVFAIL when queried from our recdns stems from the fact that it forwards the request to 10.3.0.1, which sends it back to coredns, and loops.

Ideally it would be configured for the specific zones it should know about - like "28.67.10.in-addr.arpa" - and then we could disable the fallthrough and let it return NXDOMAIN if it gets asked for something that isn't there. But overall I don't think that is the important bit to fix, just documenting the reason for the reported error.

It might be possible to work around this by creating headless services for these jobs that access the databases -- definitely worth trying.

@Clement_Goubert @JMeybohm Does this sound crazypants to you?

It might be possible to work around this by creating headless services for these jobs that access the databases -- definitely worth trying.

@Clement_Goubert @JMeybohm Does this sound crazypants to you?

No I think it makes sense, but we need to document that headless services are just for DNS purposes and not to be used to connect to services.

It might be possible to work around this by creating headless services for these jobs that access the databases -- definitely worth trying.

@Clement_Goubert @JMeybohm Does this sound crazypants to you?

Not completely. But I can imagine this might come up again in X time because another thing came along that does not have the dummy headless service. Because of this I'm also not opposed to testing kubepods which would solve this in a more generic way (I'm thinking of maintenance and other k8s jobs rn). Since we build CoreDNS from source anyways it should not be too hard to give this a try.

Not completely. But I can imagine this might come up again in X time because another thing came along that does not have the dummy headless service. Because of this I'm also not opposed to testing kubepods which would solve this in a more generic way (I'm thinking of maintenance and other k8s jobs rn). Since we build CoreDNS from source anyways it should not be too hard to give this a try.

OK, I'm game for that. It's just golang; how hard could it be?

We ran into this issue again today whilst troubleshooting a persistent connection that was preventing some maintenance being done, would it be possible to prioritize this issue? Thanks!

Change #1321574 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/docker-images/production-images@master] Add coredns 1.12

https://gerrit.wikimedia.org/r/1321574

Change #1321573 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/docker-images/production-images@master] coredns: Rename coredns to coredns1.11 to support multiple versions

https://gerrit.wikimedia.org/r/1321573

Change #1327115 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] coredns: Update to 1.12.1, add kubepods support

https://gerrit.wikimedia.org/r/1327115

Change #1321573 merged by JMeybohm:

[operations/docker-images/production-images@master] coredns: Rename coredns to coredns1.11 to support multiple versions

https://gerrit.wikimedia.org/r/1321573

Change #1321574 merged by JMeybohm:

[operations/docker-images/production-images@master] Add coredns 1.12

https://gerrit.wikimedia.org/r/1321574

Change #1327115 merged by jenkins-bot:

[operations/deployment-charts@master] coredns: Update to 1.12.1, add kubepods support

https://gerrit.wikimedia.org/r/1327115

With kubepods enabled we get a .pod.cluster.local. response for reverse lookup of pod IPs (with and without service/endpoint):

jayme@deploy1003:~# host 10.192.64.108
108.64.192.10.in-addr.arpa domain name pointer citoid-staging-6d9cc444bf-6fppl.citoid.pod.cluster.local.
jayme@deploy1003:~# host 10.192.64.103
103.64.192.10.in-addr.arpa domain name pointer k8s-controller-sidecars-7f856c79c5-qjm4v.sidecar-controller.pod.cluster.local.

This is a slight behavior change given we've responded with .svc.cluster.local. names in the past (for pods reachable via a service) but I don't think that's a problem. I'll roll this out to both wikikube staging clusters for now to see if it has an impact on coredns resource usage/stability before deploying to production.

JMeybohm moved this task from Inbox to In Progress on the ServiceOps board.

Change #1327564 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Enable CoreDNS kubepods plugin in staging-eqiad

https://gerrit.wikimedia.org/r/1327564

Thanks a lot - this is going to be SUPER helpful and will allow us to troubleshoot again wikiadmin maintenance scripts that misbehave.

Change #1327564 merged by JMeybohm:

[operations/deployment-charts@master] Enable CoreDNS kubepods plugin in staging-eqiad

https://gerrit.wikimedia.org/r/1327564

Change #1327852 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] staging-codfw: Update to coredns 1.12

https://gerrit.wikimedia.org/r/1327852

Change #1327852 merged by JMeybohm:

[operations/deployment-charts@master] staging-codfw: Update to coredns 1.12

https://gerrit.wikimedia.org/r/1327852

Change #1328565 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] coredns: Enable kubepods plugin on all clusters

https://gerrit.wikimedia.org/r/1328565

Change #1328565 merged by JMeybohm:

[operations/deployment-charts@master] coredns: Enable kubepods plugin on all clusters

https://gerrit.wikimedia.org/r/1328565

The kubepods plugin is now enabled on all clusters

Change #1328675 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] coredns: Double memory limit in codfw and eqiad

https://gerrit.wikimedia.org/r/1328675

Change #1328677 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] ml-serve: Bump coredns resources

https://gerrit.wikimedia.org/r/1328677

On wikikube production the kubepods plugin makes coredns consume ~4x memory - which was not visible in staging (probably due to the limited number of pods). But it's also not super consistent when comparing datacenters or other, way smaller clusters.

I'm bumping memory limits for wikikube to 1Gi per replica and ml-server to 500Mi (which was at a very conservative 175Mi before).

Change #1328675 merged by JMeybohm:

[operations/deployment-charts@master] coredns: Double memory limit in codfw and eqiad

https://gerrit.wikimedia.org/r/1328675

Change #1328677 merged by JMeybohm:

[operations/deployment-charts@master] ml-serve: Bump coredns resources

https://gerrit.wikimedia.org/r/1328677