Page MenuHomePhabricator

spicerack dnsdisc.Discovery attempts to query depooled/disabled dns auth servers
Closed, ResolvedPublic

Description

Traffic moves AuthDNS servers in and out of rotation by editing hiera (example patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/889270).

However, Spicerack retrieves the list of all AuthDNSes via Cumin alias A:dns-auth, which itself comes from Puppet resources (query P{R:Class = profile::dns::auth})

This leads to cookbooks like sre.discovery.service-route failing whenever Traffic is doing maintenance on an AuthDNS (e.g. reimaging):
https://www.irccloud.com/pastebin/X8NtQhNI/

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 889568 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] P:spicerack: Add discovery/authdns.yaml config

https://gerrit.wikimedia.org/r/889568

Change 889601 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/software/spicerack@master] Use yaml config file for authdns

https://gerrit.wikimedia.org/r/889601

I've amended the above patch to simplify it a bit more given there is no more the need of passing a RemoteHosts instance around.

Change 889568 merged by Clément Goubert:

[operations/puppet@production] P:spicerack: Add discovery/authdns.yaml config

https://gerrit.wikimedia.org/r/889568

Change 889601 merged by jenkins-bot:

[operations/software/spicerack@master] spicerack: get authdns servers from config file

https://gerrit.wikimedia.org/r/889601

Volans claimed this task.
Volans triaged this task as Medium priority.

Spicerack v6.2.1 was deployed with the above fix (see CHANGELOG).
From my tests all seems to work fine, we'll need to double check once there is a change in the hiera config for the active authdns hosts.
@ssingh feel free to ping me when you plan to do the next reimage.
I'm boldly resolving it for now, feel free to reopen if you encounter any issue.