related to the DNS work on labs i would suspect.

https://phabricator.wikimedia.org/T72076#1059041 or related

the timeframe you describe when it stopped working and the last comment/merge there roughly match

@coren ^ is that possible ?

I don't think so because that was merged earlier.

But on March 6th https://gerrit.wikimedia.org/r/#/c/194858/ was merged; however a):

scfc@tools-login:~$ cat /etc/resolv.conf; host saucelabs.com
## THIS FILE IS MANAGED BY PUPPET
##
## source: modules/base/resolv.conf.labs.erb
## from:   base::resolving

domain eqiad.wmflabs
search eqiad.wmflabs labs.eqiad.wmnet
options timeout:5 ndots:2
nameserver 10.68.16.1
saucelabs.com has address 162.222.73.28
saucelabs.com mail is handled by 5 ALT1.ASPMX.L.GOOGLE.com.
saucelabs.com mail is handled by 10 ASPMX3.GOOGLEMAIL.com.
saucelabs.com mail is handled by 1 ASPMX.L.GOOGLE.com.
saucelabs.com mail is handled by 5 ALT2.ASPMX.L.GOOGLE.com.
saucelabs.com mail is handled by 10 ASPMX2.GOOGLEMAIL.com.
scfc@tools-login:~$

and b) if that change would have caused DNS failures, it would be very strange if that would have been limited to saucelabs.com, and I haven't heard more complains about DNS failures in general after that change got merged.

The only net effect the change can make is that iff the fqdn has exactly one dot (as is the case here) then the local domains would be checked before (but not instead of) the DNS roots. It can't make a name that once resolved no longer do so.

I found the root cause to be this option in /etc/resolv.conf

options timeout:5 ndots:2

specifically it's the "ndots" thing. if i comment that out, it works:

 host saucelabs.com
saucelabs.com has address 162.222.73.28

in combination with

domain eqiad.wmflabs
search eqiad.wmflabs labs.eqiad.wmnet

it is the reason why it is appending eqiad.wmflabs.

This is a puppetized file though, so needs a patch.

snippet from man page of resolv.conf

options ndots:n below to avoid man-in-the-middle attacks and unnecessary traffic for the root-dns-servers. Note that this process may be slow and will generate a lot of network traffic if the servers for the listed domains are not local, and that queries will time out if no server is available for one of the domains.

another option is if you just replace saucelabs.com with www.saucelabs.com , it is the same IP and works without changes.

Change 196731 had a related patch set uploaded (by Dzahn):
don't use 'ndots: 2' in labs resolv.conf

https://gerrit.wikimedia.org/r/196731

ndots:2 is necessary for something else, the actual bug is that the dnsmasq server should emphathically not respond with SERVFAIL allowing resolution to proceed.

*headdesk* *mumble, mumble, dnsmasq*

Can you use www.saucelabs.com as a workaround? I'm going to bump up the priority of having a proper DNS server for Labs that is anything but dnsmasq.

The error doesn't seem to lie with dnsmasq. On tools-login, the look-up succeeds, on tools-trusty, it fails. So it seems to be related to some change in Ubuntu Trusty.

No, it's just that the Precise libresolv seems to be a little more forgiving and skips over the SERVFAIL - which just hides the actual issue that dnsmasq really shouldn't be returning a SERVFAIL at all in the first place.

To wit:

marc@tools-trusty:~$ host notexist
Host notexist.eqiad.wmflabs not found: 2(SERVFAIL)

Which is clearly incorrect, and happens because:

marc@tools-login:~$ dig @10.68.16.1 notexist.eqiad.wmflabs

; <<>> DiG 9.8.1-P1 <<>> @10.68.16.1 notexist.eqiad.wmflabs
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 2875
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
[...]

Which is completely broken.

I suspect this error got introduced when I switched over the CI pool from the Trusty instances we created October/December 2014 to the re-created ones from this month.

@coren: But there are you querying the Labs server, and (I think) dnsmasq just passes the request upstream:

scfc@tools-login:~$ dig @10.68.16.1 notexist.eqiad.wmflabs

; <<>> DiG 9.8.1-P1 <<>> @10.68.16.1 notexist.eqiad.wmflabs
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 1760
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;notexist.eqiad.wmflabs.                IN      A

;; Query time: 1447 msec
;; SERVER: 10.68.16.1#53(10.68.16.1)
;; WHEN: Sat Mar 14 01:33:57 2015
;; MSG SIZE  rcvd: 40

scfc@tools-login:~$ dig @10.68.16.1 notexist.eqiad.wmflabs.de

; <<>> DiG 9.8.1-P1 <<>> @10.68.16.1 notexist.eqiad.wmflabs.de
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 54424
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;notexist.eqiad.wmflabs.de.     IN      A

;; AUTHORITY SECTION:
de.                     3600    IN      SOA     f.nic.de. its.denic.de. 2015031405 7200 7200 3600000 7200

;; Query time: 2 msec
;; SERVER: 10.68.16.1#53(10.68.16.1)
;; WHEN: Sat Mar 14 01:34:09 2015
;; MSG SIZE  rcvd: 95

scfc@tools-login:~$

So (from a distance) it appears as if the WMF nameserver that dnsmasq is referring to for .wmflabs is returning SERVFAIL.

And:

scfc@tools-login:~$ dig @10.68.16.1 tools-login.eqiad.wmflabs

; <<>> DiG 9.8.1-P1 <<>> @10.68.16.1 tools-login.eqiad.wmflabs
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37165
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tools-login.eqiad.wmflabs.     IN      A

;; ANSWER SECTION:
tools-login.eqiad.wmflabs. 300  IN      A       10.68.16.7

;; Query time: 0 msec
;; SERVER: 10.68.16.1#53(10.68.16.1)
;; WHEN: Sat Mar 14 01:37:41 2015
;; MSG SIZE  rcvd: 59

scfc@tools-login:~$

Could it be that .eqiad.wmflabs is configured to query LDAP/OpenStack, and if it is passed a name with a dot in it, that causes a failure => SERVFAIL?

(Or a host name that does not exist.)

In T92351#1118579, @scfc wrote:

So (from a distance) it appears as if the WMF nameserver that dnsmasq is referring to for .wmflabs is returning SERVFAIL.

It's supposed to be authoritative for it. :-)

http://www.linuxquestions.org/questions/linux-networking-3/powerdns-servfail-945615/ (NB: MySQL) suggests that an SOA record in LDAP may be missing; AFAICS, default-soa-name is set to labs-ns0.wikimedia.org.

about workarounds:

/etc/nsswitch says: hosts: files dns
so to first check files and then dns

if we add saucelabs.com to /etc/hosts:

162.222.73.28 saucelabs.com

that will not fix it with the host command or dig because they will ask DNS directly anyways, but anything using gethostbyname() should use the entry from the file. would that unbreak the oojs-core-npm gate job for now?

getent ahosts saucelabs.com
162.222.73.28   STREAM saucelabs.com

what also works: adding a trailing dot:

root@integration-slave1401:~# host saucelabs.com.
saucelabs.com has address 162.222.73.28

but i guess that is the same problem as with adding www.

Change 196775 had a related patch set uploaded (by Yuvipanda):
labs: set resolf.conf ndots to 1

https://gerrit.wikimedia.org/r/196775

Change 196775 abandoned by Yuvipanda:
labs: set resolf.conf ndots to 1

https://gerrit.wikimedia.org/r/196775

Has someone looked at whether there is an SOA record in LDAP? If that is the source of the problem, fixing it would be much easier than working around it with resolv.conf.

Interestingly, dig saucelabs.com on tools-trusty works fine.

@scfc The SOA records are there; though it's not immediately clear that they work properly either way.

@yuvipanda: That's not so much "interesting" as "expected". Dig ignores the search order so it never asks dnsmasq for a name that doesn't exist and never gets SERVFAIL. :-)

So what does dig notexist.eqiad.wmflabs return on the server where dnsmasq is running?

If I query labs-ns0/labs-ns1 externally, they return NXDOMAIN, e. g.:

[tim@passepartout ~]$ dig @labs-ns0.wikimedia.org notexist.eqiad.wmflabs                                                                                     
                                                                                                                                                             
; <<>> DiG 9.9.3-rl.13207.22-P2-RedHat-9.9.3-16.P2.fc19 <<>> @labs-ns0.wikimedia.org notexist.eqiad.wmflabs
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 6277
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;notexist.eqiad.wmflabs.                IN      A

;; AUTHORITY SECTION:
eqiad.wmflabs.          3600    IN      SOA     virt1000.wikimedia.org. hostmaster.wikimedia.org. 1426372504 1800 3600 86400 7200

;; Query time: 148 msec
;; SERVER: 208.80.154.19#53(208.80.154.19)
;; WHEN: Mo Mär 16 14:49:39 UTC 2015
;; MSG SIZE  rcvd: 109

[tim@passepartout ~]$

Change 196731 abandoned by Dzahn:
don't use 'ndots: 2' in labs resolv.conf

https://gerrit.wikimedia.org/r/196731

That may still be an option, once he have at least one that actually works right. :-)

Change 196731 restored by Hashar:
don't use 'ndots: 2' in labs resolv.conf

https://gerrit.wikimedia.org/r/196731

On CI we do DNS requests for pubic DNS entry so we apparently need to remove the ndots:2 option. Hence @Krinkle reapplied https://gerrit.wikimedia.org/r/196731 on the integration puppet master.

AFAICT, this problem solved itself (as expected) since we switched to a properly functionning DNS server.

Too bad for the readers Google will bring here in the future: Nearly four months of investigation, no explanation why dnsmasq (allegedly) misbehaves, no link to a bug report upstream, just "use a properly functioning DNS server" as take-away. But probably worse for the developers whose free software we were using successfully for so long.

Change 196731 abandoned by Dzahn:
base: Don't use 'ndots: 2' in labs resolv.conf

https://gerrit.wikimedia.org/r/196731

We still have that the Gerrit patch applied on the integration labs project. Filled T105297 to get it removed and thus reenable ndots: 2.

In T92351#1439472, @scfc wrote:

Too bad for the readers Google will bring here in the future: Nearly four months of investigation, no explanation why dnsmasq (allegedly) misbehaves, no link to a bug report upstream, just "use a properly functioning DNS server" as take-away. But probably worse for the developers whose free software we were using successfully for so long.

Your comment is not helping anyone. We long wanted to drop dnsmasq because it has a bunch of other issue and for the purpose of this bug the workaround was quiet easily : just remove ndots:2.

Overall this issue and others prompted ops to finally migrate out of dnsmasq to a better DNS system/architecture for labs. So that is a net win for us. I don't think it was worth our time to investigate a software we already planned to drop entirely.

I don't mind if ops wouldn't have investigated the issue. But doing so, claiming that dnsmasq is at fault, but not offering any explanation or chance for developers to fix a (perceived) error is slanderous to me.

So I do hope that my comment is helpful so that for example when someone finds an (alleged) error in one of your scripts and just deletes it with the comment "@hashar wrote it", you know that outside the bubble you won't be harassed.

In T92351#1441395, @scfc wrote:

But doing so, claiming that dnsmasq is at fault, but not offering any explanation or chance for developers to fix a (perceived) error is slanderous to me.

We did enough debugging to determine without a doubt that dnsmasq returned SRVFAIL rather than NXDOMAIN for entries which it did not know of over a domain it claimed it was authoritative for. This is incorrect behaviour.

We did not investigate the cause of that incorrect behaviour because dnsmasq was already causing a number of other (unrelated) issues, and it was known that it would be discareded in the short term and replaced. Spending time debugging a system you know is on its way out is - at best - futile when a workaround exists.