Whilst troubleshooting a database issue, I found the following IP (being active) on show processlist: 10.67.28.73
When checking which host this was, in order to ssh and debug locally I ran into:
marostegui@cumin1003:~$ host 10.67.28.73 Host 73.28.67.10.in-addr.arpa not found: 2(SERVFAIL)
The following IRC conversation happened:
[11:26:20] <marostegui> elukey is this normal? [11:26:21] <marostegui> [09:25:57] marostegui@cumin1003:~$ host 10.67.28.73 [11:26:21] <marostegui> Host 73.28.67.10.in-addr.arpa not found: 2(SERVFAIL) [11:27:55] <elukey> marostegui: lovely, never seen it before. Does it happen only with that IP, or more? [11:28:21] <marostegui> elukey: I was debugging something and found that it, I just tried with another host and it does look fine [11:31:13] <elukey> marostegui: but how did you come up with the IP? Is it appearing in some log etc..? Seems a k8s one in theory, so I am wondering if IPs belonging to dead pods may end up with servfail and not nxdomain for $reason [11:31:28] <marostegui> elukey: yeah, it was from a show processlist in a database [11:31:36] <marostegui> And I wanted to check which host it was [11:31:38] <marostegui> To ssh there etc [11:32:39] <elukey> I tried other stuff like 10.67.28.71 etc.. and they end up in servfail, so I am almost sure the IP you found was related to a pod that got killed [11:33:33] <marostegui> elukey: but that IP is actively doing things eh [11:35:45] <elukey> marostegui: oh ok I didn't get this bit before [11:36:15] <elukey> mmmm maybe it is related to coredns on the target k8s cluster [11:37:00] <elukey> netbox suggests that it is a dse k8s pod https://netbox.wikimedia.org/ipam/prefixes/538/ [11:37:16] <marostegui> elukey: The thing is a bit tricky, because there's an IP doing stuff but I cannot access it as it doesn't resolve a name [11:37:43] <elukey> marostegui: yeah I know, lemme try to find more info about the IP. [11:37:54] <marostegui> elukey: <3 [11:43:52] <elukey> wikidatawiki-sql-xml-wikidatawiki-dump-remaining-full-ev2anki running on dse-k8s-worker1014.eqiad.wmnet