Page MenuHomePhabricator

Wikimedia DNS somehow unable to resolve WMCS hosted domain names
Open, Needs TriagePublic

Description

At the moment at least doh3005 seems to be unable to resolve names for WMCS hosted names, instead returning a mixture of 502s (for the DoH endpoint) and SERVFAILs:

1taavi@runko:~ $ kdig +https restricted.bastion.wmcloud.org @185.71.138.138
2;; HTTP session (HTTP/2-POST)-(185.71.138.138/dns-query)-(status: 502)
3;; WARNING: can't receive reply from 185.71.138.138@443(HTTPS)
4;; ERROR: failed to query server 185.71.138.138@443(TCP)
5taavi@runko:~ $ kdig +tls restricted.bastion.wmcloud.org @185.71.138.138
6;; TLS session (TLS1.3)-(ECDHE-X25519)-(ECDSA-SECP256R1-SHA256)-(AES-128-GCM)
7;; ->>HEADER<<- opcode: QUERY; status: SERVFAIL; id: 43454
8;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 1
9
10;; EDNS PSEUDOSECTION:
11;; Version: 0; flags: ; UDP size: 512 B; ext-rcode: NOERROR
12;; PADDING: 394 B
13
14;; QUESTION SECTION:
15;; restricted.bastion.wmcloud.org. IN A
16
17;; Received 457 B
18;; Time 2026-07-21 15:03:49 EEST
19;; From 185.71.138.138@853(TLS) in 107.5 ms
20taavi@runko:~ $ kdig +https restricted.bastion.wmcloud.org @185.71.138.138
21;; TLS session (TLS1.3)-(ECDHE-SECP256R1)-(ECDSA-SECP256R1-SHA256)-(AES-128-GCM)
22;; HTTP session (HTTP/2-POST)-(185.71.138.138/dns-query)-(status: 200)
23;; ->>HEADER<<- opcode: QUERY; status: SERVFAIL; id: 0
24;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 1
25
26;; EDNS PSEUDOSECTION:
27;; Version: 0; flags: ; UDP size: 512 B; ext-rcode: NOERROR
28;; PADDING: 394 B
29
30;; QUESTION SECTION:
31;; restricted.bastion.wmcloud.org. IN A
32
33;; Received 457 B
34;; Time 2026-07-21 15:03:58 EEST
35;; From 185.71.138.138@443(HTTPS) in 104.2 ms
36taavi@runko:~ $ kdig +https restricted.bastion.wmcloud.org @185.71.138.138^C
37taavi@runko:~ $ kdig +tls +nsid restricted.bastion.wmcloud.org @185.71.138.138
38;; TLS session (TLS1.3)-(ECDHE-X25519)-(ECDSA-SECP256R1-SHA256)-(AES-128-GCM)
39;; ->>HEADER<<- opcode: QUERY; status: SERVFAIL; id: 3765
40;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 1
41
42;; EDNS PSEUDOSECTION:
43;; Version: 0; flags: ; UDP size: 512 B; ext-rcode: NOERROR
44;; NSID: 646F6833303035 "doh3005"
45;; PADDING: 394 B
46
47;; QUESTION SECTION:
48;; restricted.bastion.wmcloud.org. IN A
49
50;; Received 468 B
51;; Time 2026-07-21 15:04:20 EEST
52;; From 185.71.138.138@853(TLS) in 111.4 ms

Related Objects

Event Timeline

Thanks for reporting. It is a bit weird but now if I test it against doh3005, it seems to have resolved. Can you confirm?

Not that we have any logs around queries but at least looking at the SERVFAILs in pdns-rec's journal, there seem to be no matches for this domain, which doesn't help much.

Indeed, works again for me. Is there something I could've done when it was broken that would've given you more info to work off of?

Indeed, works again for me. Is there something I could've done when it was broken that would've given you more info to work off of?

It's a good question that deserves more thought but here's what I do for now:

  • see if you can connect to another host; you can query a Wikidough host directly as well, so you can do kdig +tls restricted.bastion.wmcloud.org @208.80.154.70 instead of the Wikidough anycast and see if that helps
  • we don't have query logs in the dnsdist interface as we disabled them during compile-time, so the most you can do is to SSH to the Wikidough host that is giving you the error and tail the pdns-recursor logs to see if you get a reason for the SRVFAIL
  • try another domain for the particular host, to rule out any domain-specific issues

Let me know your thoughts if any.

pdns-recursor on Wikidough hosts is running on port 53, so you can also do a local query to see if it is something with dnsdist, or pdns-rec. dig @127.0.0.1 <domain> should give you the result from pdns-rec and not dnsdist.