With Anycast recdns fully deployed for some time now, the traffic to LVS recdns has dropped off substantially. Quick checks show only healthcheck monitoring and a few queries from some hardware devices like PDUs to clean up. This task is to track down and eliminate the remaining few cases and the decom of these service IPs and associated LVS configuration, etc.
In a sample I just took across all recdns for a little over 15 minutes of sniffer time looking for requests to the legacy LVS-based recdns IPs:
- ulsfo, eqsin, and esams had no traffic to them at all (yay! and makes basic sense)
- eqiad had a handful of requests from:
- codfw had more-interesting traffic from:
The PDUs I kind of expected. IIRC some of them can't be updated easily, and honestly they're not a huge problem. Will dig a bit more on those other cases!
Dug into the odd cases from install2002 and kraz - the common pattern here is that there are some daemons in the world which both (a) parse /etc/resolv.conf for themselves because they use their own custom DNS client code and (b) don't ever re-read that file if it changes. A few of those are daemons we actually use, which happen to have not had their daemon (or the host) restarted since our resolv.conf was switched to the new recdns IP a few months ago (~Aug-Sept timeframe, it was rolled out at different times to different places).
In these particular cases, install2002 needed a squid3 daemon restart (done), and for the kraz case it's ircd (which is an old version of ircd-ratbox used for mw_rc_irc stuff (which I haven't restarted, because I'm not sure how fragile that stuff is)).
Next week I might do a much longer sniff (hours), and see if I can find any more such edge cases.
Status: The actual LVS portion of this is now completely removed globally. The IP addresses themselves are also completely unconfigured and removed from service at the all the edge sites, but not the core ones. What remains is that the legacy LVS recdns IPs 220.127.116.11 (eqiad) and 18.104.22.168 (codfw) are still statically-configured to avoid breaking any of the leftover dependencies on these IPs. Sniffer monitoring has shown at least the ircd instance on kraz is still using outdated resolv.conf data and hitting these IPs, several hardware PDUs are using them as well, and there are possibly other such cases which are rarer and thus harder to observe in short samples (I've done up to 1h samples).
The static (as in non-LVS) configuration of these is puppetized, and the eqiad and codfw core routers have explicit static routes sending 22.214.171.124 to dns1002 and 126.96.36.199 to dns2002 (the 01 boxes are also acceptable backup targets if necessary). The routes in the juniper configs are tagged with a comment referencing this ticket.
Once we're sure we're ready to destroy these last remnants of service (after the holidays! and investigating the remaining PDUs situation and kraz and taking longer sniffs), what remains to finish decomming these and close this ticket up is:
- Remove the manual routes referenced above from cr-(eqiad|codfw)
- Merge and deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/556178/ (carefully one at a time on dns00), removing the service IP listeners and IP address defs)
- Merge and deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/556179/ (doesn't have to be as careful, just cleans up the bits that remain after the above in the puppet sense)
- Merge and deploy the DNS patch https://gerrit.wikimedia.org/r/#/c/operations/dns/+/556230/ (removes the last comment lines noting that these IPs are still in use)
The kraz case is gone now (yay!) and hasn't recurred since the ircd restart above. What's left appears to be all infrastructure stuff: PDUs, switches, firewalls, etc. I've picked up quite a few of them in a few hours, so I'm going to let it run for a full 24h to try to capture them all, and then I'll make some sub-tasks to clean them up.
@BBlack FYI the step:
Merge and deploy the DNS patch https://gerrit.wikimedia.org/r/#/c/operations/dns/+/556230/ (removes the last comment lines noting that these IPs are still in use)
Has been done with the migration to Netbox's autogenerated data. If the IPs with final octet 254 should be reserved, they can be marked as such in Netbox.
The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!
These IPs came up the other day in an IRC conversation. They do still exist in dns00 puppetized live config. They're both still intended to still be functional, but they're lacking some accountability between the DNS repo and netbox. I went ahead and removed the last DNS repo reference to them, and I'm adding DNS names to them in netbox (and referencing this ticket), so that clears up the accounting issues a bit.
Before we can remove them from service (from router config, dns host config, and netbox), we have to eliminate the remaining device configs which are still relying on these IPs. The last hosts (e.g. kraz) were fixed a while back, but there are still infra devices using these IPs for DNS resolution (PDUs, switches, SRXs, etc). Haven't sniffed for what the remaining cases are lately, so should probably re-check and see how widespread the problem is anymore.
FTR - I did a quick 1-hour capture on these today just to see whether there was any sign of remaining cases, and there still are some. Probably a 24+ hour capture would get more of them, but the few that showed up in an hour today were, ps1-a5-codfw, ps1-d3-codfw, ps1-b2-codfw, ps1-22-ulsfo. Should probably at some point (no rush!) just audit the PDUs and other bits to see which still haven't been converted to using 10.3.0.1 for recdns.