The Nodepool instances receives their DNS configuration from DHCP. For a Jessie one /etc/resolv.conf is:

domain eqiad.wmflabs
search eqiad.wmflabs
nameserver 208.80.155.118

That IP is labs-recursor0.wikimedia.org. I guess it has/had some troubles.

Occurred again https://integration.wikimedia.org/ci/job/operations-puppet-tox-py27-jessie/10238/console again on a Jessie node on a Nodepool instance:

00:00:28.375 fatal: unable to access 'https://gerrit.wikimedia.org/r/operations/puppet/mariadb/': Could not resolve host: gerrit.wikimedia.org

Is there a way to re-trigger the CI jobs other than rebasing the entire patch in Gerrit?

In T137460#2376125, @MoritzMuehlenhoff wrote:

Is there a way to re-trigger the CI jobs other than rebasing the entire patch in Gerrit?

Yup on the Gerrit change you can comment recheck which will enqueue the patchset again as if it has been just created.

Worth adding to Scrum of Scrums?

I'm wondering if this is only happening on nodepool.

The resolv.conf for different instances:

Nodepool Trusty:

nameserver 208.80.155.118
search eqiad.wmflabs

Nodepool Jessie

domain eqiad.wmflabs
search eqiad.wmflabs
nameserver 208.80.155.118

Permanent slaves using the whole puppet.git provisioning have:

domain integration.eqiad.wmflabs
search integration.eqiad.wmflabs eqiad.wmflabs
nameserver 208.80.155.118
nameserver 208.80.154.20
options timeout:2 ndots:2

I have looked at a Jessie Nodepool instance. It has two dhclient running:

dhclient -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0

The lease:

lease {
  interface "eth0";
  fixed-address 10.68.16.187;
  filename "jessie-installer/pxelinux.0";
  server-name "carbon.wikimedia.org";
  option subnet-mask 255.255.248.0;
  option dhcp-lease-time 86400;
  option routers 10.68.16.1;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.68.16.1;
  option unknown-209 "pxelinux.cfg/ttyS1-115200";
  option domain-name-servers 208.80.155.118;
  option dhcp-renewal-time 43200;
  option dhcp-rebinding-time 75600;
  option broadcast-address 10.68.23.255;
  option host-name "ci-jessie-wikimedia-147750";
  option domain-name "eqiad.wmflabs";
  renew 4 2016/06/16 07:55:36;
  rebind 4 2016/06/16 17:04:15;
  expire 4 2016/06/16 20:04:15;
}

What ever DHCP server is answering would probably need the second labs resolver to be filled:

- option domain-name-servers 208.80.155.118;
+ option domain-name-servers 208.80.155.118, 208.80.154.20;

In puppet we have modules/openstack/templates/liberty/nova/dnsmasq-nova.conf.erb :

#Clients should use the designate-backed dns server rather than dnsmasq
dhcp-option=option:dns-server,<%= @recursor_ip %>

Potentially if @recursor_ip is found out to be list/array, we could sort and join with , all the entries.

Forgot, a fully commented dnsmasq.conf http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq.conf.example

And something I thought about: maybe the base Jessie image does not come with any sort of DNS caching, harming the labs recursor uselessly.

Change 325278 had a related patch set uploaded (by Alex Monk):
Send secondary DNS recursor IP from labs DHCP

https://gerrit.wikimedia.org/r/325278

Change 325278 merged by Andrew Bogott:
Send secondary DNS recursor IP from labs DHCP

https://gerrit.wikimedia.org/r/325278

This should be resolved for new CI instances. Hashar, can you verify that new VMs now have labs-ns0 and labs-ns1 IPs in resolv.conf?

Sorry I have missed updates on this task. https://gerrit.wikimedia.org/r/#/c/325278/ updated dnsmasq-nova.conf.erb

dhcp-option=option:dns-server,<%= @recursor_ip %>,<%= @recursor_secondary_ip %>

Does not reflect back on the instances (checked Trusty/Jessie instances on contintcloud, deployment-prep and integration tenants).

@Andrew can you double check the content of /etc/dnsmasq-nova.conf and maybe openstack or dnsmasq needs to be reloaded/restarted to account for the change?

That change will definitely only affect new instances, not anything that existed before I made the puppet change. Since we're talking about instances that aren't puppetized (is that right?) I'm not sure how to make them reload.

in /etc/dnsmasq-nova.conf:

dhcp-option=option:dns-server,208.80.155.118,208.80.154.20

I have created instances after that change was merged, and they only have one DNS server listed. The leases expire daily, so all instances should have picked up the second nameserver soon after merging. IMHO nova-network has not been restarted after the merge, thus dnsmasq is still using the old configuration.

Good point. I have just restarted nova-network (and killed the dnsmasq procs so they're forced to respawn). Any better now?

Change 330308 had a related patch set uploaded (by Andrew Bogott):
nova-network: Refresh service if some dependency files change.

https://gerrit.wikimedia.org/r/330308

Change 330308 merged by Andrew Bogott:
nova-network: Refresh service if config files change.

https://gerrit.wikimedia.org/r/330308