Page MenuHomePhabricator

Decom LVS recdns
Open, MediumPublic

Description

With Anycast recdns fully deployed for some time now, the traffic to LVS recdns has dropped off substantially. Quick checks show only healthcheck monitoring and a few queries from some hardware devices like PDUs to clean up. This task is to track down and eliminate the remaining few cases and the decom of these service IPs and associated LVS configuration, etc.

Event Timeline

BBlack created this task.Dec 6 2019, 2:00 PM
Restricted Application added a project: Operations. · View Herald TranscriptDec 6 2019, 2:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
BBlack moved this task from Triage to DNS Infra on the Traffic board.

Change 555520 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] lvs recdns: switch DNS aliases to anycast

https://gerrit.wikimedia.org/r/555520

Change 555537 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] lvs recdns decom

https://gerrit.wikimedia.org/r/555537

Change 555538 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] lvs recdns post-decom cleanup

https://gerrit.wikimedia.org/r/555538

Change 555539 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] lvs recdns: get rid of legacy recursor hostnames

https://gerrit.wikimedia.org/r/555539

BBlack added a comment.Dec 6 2019, 4:56 PM

In a sample I just took across all recdns for a little over 15 minutes of sniffer time looking for requests to the legacy LVS-based recdns IPs:

  • ulsfo, eqsin, and esams had no traffic to them at all (yay! and makes basic sense)
  • eqiad had a handful of requests from:
    • ps1-d7-eqiad.mgmt.eqiad.wmnet
    • ps1-d2-eqiad.mgmt.eqiad.wmnet
    • ps1-c1-eqiad.mgmt.eqiad.wmnet
  • codfw had more-interesting traffic from:
    • ps1-a8-codfw.mgmt.codfw.wmnet
    • ps1-22-ulsfo.mgmt.ulsfo.wmnet
    • install2002.wikimedia.org
    • kraz.wikimedia.org

The PDUs I kind of expected. IIRC some of them can't be updated easily, and honestly they're not a huge problem. Will dig a bit more on those other cases!

Change 555520 merged by BBlack:
[operations/dns@master] lvs recdns: switch DNS aliases to anycast

https://gerrit.wikimedia.org/r/555520

BBlack added a comment.EditedDec 6 2019, 6:16 PM

Dug into the odd cases from install2002 and kraz - the common pattern here is that there are some daemons in the world which both (a) parse /etc/resolv.conf for themselves because they use their own custom DNS client code and (b) don't ever re-read that file if it changes. A few of those are daemons we actually use, which happen to have not had their daemon (or the host) restarted since our resolv.conf was switched to the new recdns IP a few months ago (~Aug-Sept timeframe, it was rolled out at different times to different places).

In these particular cases, install2002 needed a squid3 daemon restart (done), and for the kraz case it's ircd (which is an old version of ircd-ratbox used for mw_rc_irc stuff (which I haven't restarted, because I'm not sure how fragile that stuff is)).

Next week I might do a much longer sniff (hours), and see if I can find any more such edge cases.

colewhite triaged this task as Medium priority.Dec 6 2019, 11:19 PM

Change 556177 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] lvs recdns: eqiad and codfw keep old addr, for now

https://gerrit.wikimedia.org/r/556177

Change 556178 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] lvs recdns: remove legacy IP definition, step 1

https://gerrit.wikimedia.org/r/556178

Change 556179 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] lvs recdns: remove legacy IP definition, step 2

https://gerrit.wikimedia.org/r/556179

Mentioned in SAL (#wikimedia-operations) [2019-12-10T16:23:05Z] <bblack> cr[12]-codfw: Adding static route for 208.80.153.254 (legacy lvs recdns IP) to dns2002.wikimedia.org - T239993

Mentioned in SAL (#wikimedia-operations) [2019-12-10T16:25:23Z] <bblack> cr[12]-eqiad: Adding static route for 208.80.154.254 (legacy lvs recdns IP) to dns1002.wikimedia.org - T239993

Mentioned in SAL (#wikimedia-operations) [2019-12-10T16:37:40Z] <bblack> lvs* + dns*: puppet disabled for lvs recdns decom work - T239993

Change 555537 merged by BBlack:
[operations/puppet@production] lvs recdns: decom lvs-specific parts

https://gerrit.wikimedia.org/r/555537

Change 556177 merged by BBlack:
[operations/puppet@production] lvs recdns: eqiad and codfw keep old addr, for now

https://gerrit.wikimedia.org/r/556177

Change 555538 merged by BBlack:
[operations/puppet@production] lvs recdns: clean up realserver def

https://gerrit.wikimedia.org/r/555538

Change 556230 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] lvs recnds: remove last remaining revdns comments

https://gerrit.wikimedia.org/r/556230

Change 555539 merged by BBlack:
[operations/dns@master] lvs recdns: get rid of legacy recursor hostnames

https://gerrit.wikimedia.org/r/555539

Status: The actual LVS portion of this is now completely removed globally. The IP addresses themselves are also completely unconfigured and removed from service at the all the edge sites, but not the core ones. What remains is that the legacy LVS recdns IPs 208.80.154.254 (eqiad) and 208.80.153.254 (codfw) are still statically-configured to avoid breaking any of the leftover dependencies on these IPs. Sniffer monitoring has shown at least the ircd instance on kraz is still using outdated resolv.conf data and hitting these IPs, several hardware PDUs are using them as well, and there are possibly other such cases which are rarer and thus harder to observe in short samples (I've done up to 1h samples).

The static (as in non-LVS) configuration of these is puppetized, and the eqiad and codfw core routers have explicit static routes sending 208.80.154.254 to dns1002 and 208.80.153.254 to dns2002 (the 01 boxes are also acceptable backup targets if necessary). The routes in the juniper configs are tagged with a comment referencing this ticket.

Once we're sure we're ready to destroy these last remnants of service (after the holidays! and investigating the remaining PDUs situation and kraz and taking longer sniffs), what remains to finish decomming these and close this ticket up is:

jcrespo moved this task from Backlog to Acknowledged on the Operations board.Dec 11 2019, 5:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-20T16:53:05Z] <bblack> kraz.wikimedia.org ( https://wikitech.wikimedia.org/wiki/IRCD ) - stopping ircecho then ircd, then restarting them in reverse order - T239993

The kraz case is gone now (yay!) and hasn't recurred since the ircd restart above. What's left appears to be all infrastructure stuff: PDUs, switches, firewalls, etc. I've picked up quite a few of them in a few hours, so I'm going to let it run for a full 24h to try to capture them all, and then I'll make some sub-tasks to clean them up.