Traffic moves AuthDNS servers in and out of rotation by editing hiera (example patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/889270).
However, Spicerack retrieves the list of all AuthDNSes via Cumin alias A:dns-auth, which itself comes from Puppet resources (query P{R:Class = profile::dns::auth})
This leads to cookbooks like sre.discovery.service-route failing whenever Traffic is doing maintenance on an AuthDNS (e.g. reimaging):
https://www.irccloud.com/pastebin/X8NtQhNI/
Change 889568 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/puppet@production] P:spicerack: Add discovery/authdns.yaml config
Change 889601 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/software/spicerack@master] Use yaml config file for authdns
I've amended the above patch to simplify it a bit more given there is no more the need of passing a RemoteHosts instance around.
Change 889568 merged by Clément Goubert:
[operations/puppet@production] P:spicerack: Add discovery/authdns.yaml config
Change 889601 merged by jenkins-bot:
[operations/software/spicerack@master] spicerack: get authdns servers from config file
Spicerack v6.2.1 was deployed with the above fix (see CHANGELOG).
From my tests all seems to work fine, we'll need to double check once there is a change in the hiera config for the active authdns hosts.
@ssingh feel free to ping me when you plan to do the next reimage.
I'm boldly resolving it for now, feel free to reopen if you encounter any issue.