That is caused by puppet change https://gerrit.wikimedia.org/r/#/c/202278/ which renames the variable from use_dnsmasq to use_dnsmasq_server. Since it is not set in hiera for the integration project, all instances magically migrated to the new resolver :(

I have set in the hiera configuration:

use_dnsmasq: true
use_dnsmasq_server: true

I guess all puppet client certificates need to regenerated again :((((

So the problem is VERY nasty. All instances have been switched to the new DNS resolver which causes puppet client to fail.

The puppetmaster is f*** up as well, once it is repaired we need on each machine to MANUALLY revert back by editing /etc/resolv.conf:

echo 'domain eqiad.wmflabs                                                                 
search eqiad.wmflabs                                                                   
nameserver 10.68.16.1' > /etc/resolv.conf
/etc/init.d/nscd restart

Since some instances have been migrated to a puppetmaster named with the labs project name inserted (integration-puppetmaster.integration.eqiad.wmflabs) the certificate is no more valid. Need to revoke it on both client and server and regenerate one.

I think that certificate signing nowadays is handled automatically, so you only need two Puppet runs patience? Also, /etc/resolv.conf should automatically revert back when use_dnsmasq_server is true (again).

I don't think this was actually caused by that patch; I suspect that it's another example of https://phabricator.wikimedia.org/T95240 which the referenced patch may or may not have fixed adequately.

@hashar we had the same problem yesterday in the staging project. The hiera hack should have been a no-op on every environment unless you had use_dnsmasq set to anything other than True in hiera. If use_dnsmasq was not set in hiera, it should have fallen back to the top-level variable set in ldap.

The variable has_dnsmasq is set in ldap. For instance, on integration-slave-trusty-1005

root@staging-palladium:~# ldapsearch -LLL -x -D 'cn=proxyagent,ou=profile,dc=wikimedia,dc=org' -w $(grep -Po "(?<=bindpw).*" /etc/ldap.conf) -b 'ou=hosts,dc=wikimedia,dc=org' "associatedDomain=integration-slave-trusty-1005*"
...
puppetVar: use_dnsmasq=true

The puppet change, $use_dnsmasq_server = hiera('use_dnsmasq', $::use_dnsmasq) should use the hiera variable use_dnsmasq unless it's not set, in which case it uses the top level variable $::use_dnsmasq which is set in ldap.

@Andrew determined that in staging (and possibly here) $::use_dnsmasq == 'true' which was causing our problem because $::use_dnsmasq needed to be equal to the boolean True which is why we needed the hiera hack.

As to why the variable name changed, the puppet DSL explicitly disallows reassignment: https://docs.puppetlabs.com/puppet/latest/reference/lang_variables.html#no-reassignment

I have filled T95288: Designate should support split horizon resolution to yield private IP of instances behind a public DNS entry

Poked @Andrew about the issue on https://lists.wikimedia.org/pipermail/labs-l/2015-April/003592.html

The puppet failure where due to the hostname of the puppetmaster changing. That causes puppetmaster self to no more recognize the master as being the master and alter the puppet.conf to remove the [master] section. The puppetmaster process is still around though, it ends up with the SSL cert of the client which is in [main] section.

Beta ends up with the same issue T95586