Page MenuHomePhabricator
Create Task
Maniphest T186550

Anycast recdns
Closed, ResolvedPublic
Actions

Description

The primary, titular goal here is to increase our reliability and kill some resolv.conf config complexity by having internal recdns anycast IPs that are reliable across our network. This also gets us some experience relevant to the (later) task of doing public anycast AuthDNS (parent task).

The simplest way to do this would be to add them as LVS public IPs for the existing LVS anycast services. LVS would still monitor backend pdns_recursor health, and handle BGP advertisements to the routers.

Another route we've been exploring here, however, is to also take LVS out of the picture and have the recdns servers self-monitor and advertise the anycast IP directly. There and up- and down- side tradeoffs to that design decision to be discussed!

Wiki on specifics: https://wikitech.wikimedia.org/wiki/Anycast_recursive_DNS

Details

SubjectRepoBranchLines +/-
bird: 10s delay after route withdrawoperations/puppetproduction+5 -2
anycast: bind to real serviceoperations/puppetproduction+6 -6
anycast healthchecker should be able to bindoperations/puppetproduction+12 -3
Bird anycast, add monitoring for anycast-healthcheckeroperations/puppetproduction+209 -0
Reserve 10.3.0.0/30 for recdns anycast (backup static route)operations/dnsmaster+1 -0
anycast recdns: test via resolv.conf on 5 hostsoperations/puppetproduction+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

BBlack triaged this task as Medium priority.Feb 5 2018, 6:33 PM
BBlack created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 5 2018, 6:33 PM
BBlack moved this task from Backlog to Some old column on the Traffic board.Feb 13 2018, 1:20 PM
BBlack mentioned this in T79058: Create HA setup for DNS recursion.Feb 13 2018, 3:05 PM
Krinkle unsubscribed.Feb 16 2018, 7:53 AM
Platonides subscribed.Mar 15 2018, 12:23 AM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:45 PM
BBlack updated the task description. (Show Details)Apr 8 2019, 2:12 PM
gerritbot added a comment.Apr 11 2019, 4:45 PM

Change 397723 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Bird anycast: add anycast_healthchecker

https://gerrit.wikimedia.org/r/397723

gerritbot added a project: Patch-For-Review.Apr 11 2019, 4:45 PM
jbond subscribed.Apr 29 2019, 4:27 PM
gerritbot added a comment.Jul 2 2019, 5:42 PM

Change 520284 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: test via resolv.conf on 5 hosts

https://gerrit.wikimedia.org/r/520284

gerritbot added a comment.Jul 3 2019, 1:23 PM

Change 520284 merged by BBlack:
[operations/puppet@production] anycast recdns: test via resolv.conf on 5 hosts

https://gerrit.wikimedia.org/r/520284

gerritbot added a comment.Jul 3 2019, 1:32 PM

Change 520441 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for LVS balancers

https://gerrit.wikimedia.org/r/520441

gerritbot added a comment.Jul 3 2019, 8:39 PM

Change 520528 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Reserve 10.3.0.0/30 for recdns anycast (backup static route)

https://gerrit.wikimedia.org/r/520528

gerritbot added a comment.Jul 3 2019, 8:55 PM

Change 520528 merged by Ayounsi:
[operations/dns@master] Reserve 10.3.0.0/30 for recdns anycast (backup static route)

https://gerrit.wikimedia.org/r/520528

Stashbot added a comment.Jul 3 2019, 8:58 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-03T20:58:58Z] <XioNoX> add static backup routes for anycast recdns on cr1/2-codfw/eqiad - T186550

gerritbot added a comment.Jul 3 2019, 10:02 PM

Change 520643 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Bird anycast, add monitoring for anycast-healthchecker

https://gerrit.wikimedia.org/r/520643

• ayounsi closed this task as Resolved.Jul 16 2019, 12:53 AM

Everything in the scope of that task is completed.

MoritzMuehlenhoff subscribed.Jul 16 2019, 6:50 AM

It's my understanding that this reduces the steps necessary to restart our recursors is now reduced to a simple depool/repool and that the previous, complex approach from
https://wikitech.wikimedia.org/wiki/Service_restarts#DNS_recursors_(in_production_and_labservices) is now obsolete, right?

• ayounsi added a comment.Jul 16 2019, 4:50 PM

It will eventually. Only a few servers are using the new IPs for now, I opened T228190 to roll it out.

BBlack added a comment.Jul 16 2019, 4:57 PM
In T186550#5335946, @MoritzMuehlenhoff wrote:

It's my understanding that this reduces the steps necessary to restart our recursors is now reduced to a simple depool/repool and that the previous, complex approach from
https://wikitech.wikimedia.org/wiki/Service_restarts#DNS_recursors_(in_production_and_labservices) is now obsolete, right?

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/520441/ (not yet deployed) will get rid of the LVS<->recdns dependency mess documented there.

gerritbot added a comment.Jul 31 2019, 10:44 PM

Change 520643 merged by Ayounsi:
[operations/puppet@production] Bird anycast, add monitoring for anycast-healthchecker

https://gerrit.wikimedia.org/r/520643

elukey reopened this task as Open.EditedAug 1 2019, 7:24 AM
elukey added a subscriber: fgiunchedi.

Couple of notes about the anycast-healthchecker:

  1. the anycast-healthchecker is not in jessie-wikimedia, so puppet on lithium/wezen is currently broken:
root@install1002:/srv/wikimedia# reprepro lsbycomponent anycast-healthchecker
anycast-healthchecker | 0.8.2-1 | stretch-wikimedia | main | amd64, i386, source
anycast-healthchecker | 0.8.2-1 |  buster-wikimedia | main | amd64, i386, source
  1. python3-docopt seems to be required by the healthchecker's nagios monitor, and it was missing on lithium. I installed it manually via apt on it and opened https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/526849/
  1. Due to 1), the anycast-healthchecker is not deployed on lithium/wezen (jessies) and in turn causes the nagios check to fire criticals in icinga. I have acked them waiting for a fix, since I am not sure what is best.
  1. https://wikitech.wikimedia.org/wiki/Anycast_recursive_DNS#Anycast_healthchecker_not_running seems not present

Cc: @fgiunchedi as FYI

fgiunchedi added a comment.Aug 1 2019, 10:17 AM

Thanks @elukey ! Indeed anycast-healthchecker isn't in jessie-wikimedia, lithium is being decom'd and if wezen gets reinstalled it'll be buster, and I installed anycast-healthchecker manually there there. No real reason though to skip jessie-wikimedia so I'll copy the packages there too.

BBlack added a subtask: T228190: Roll out Anycast RecDNS to more servers.Aug 15 2019, 2:26 PM
BBlack mentioned this in T98006: Anycast AuthDNS.Aug 15 2019, 2:48 PM
• ayounsi added a comment.Oct 15 2019, 7:53 AM
In T186550#5383508, @elukey wrote:

Couple of notes about the anycast-healthchecker:

  1. python3-docopt seems to be required by the healthchecker's nagios monitor, and it was missing on lithium. I installed it manually via apt on it and opened https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/526849/

Thanks, merged.

  1. https://wikitech.wikimedia.org/wiki/Anycast_recursive_DNS#Anycast_healthchecker_not_running seems not present

Fixed.

  1. and 3) addressed by Filippo above.
• ayounsi closed this task as Resolved.Oct 15 2019, 7:53 AM
BBlack closed subtask T228190: Roll out Anycast RecDNS to more servers as Resolved.Nov 5 2019, 5:56 PM
BBlack edited parent tasks, added: T104442: Investigate better DNS cache/lookup solutions; removed: T98006: Anycast AuthDNS.Nov 5 2019, 6:06 PM
Jgreen added a subtask: Restricted Task.Nov 14 2019, 9:26 PM
gerritbot added a comment.Dec 5 2019, 3:10 PM

Change 554866 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast healthchecker should be able to bind

https://gerrit.wikimedia.org/r/554866

gerritbot added a comment.Dec 5 2019, 3:10 PM

Change 554867 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast: bind to real service

https://gerrit.wikimedia.org/r/554867

gerritbot added a comment.Dec 5 2019, 3:10 PM

Change 554868 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] bird: 10s delay after route withdraw

https://gerrit.wikimedia.org/r/554868

gerritbot added a comment.Dec 5 2019, 3:14 PM

Change 554866 merged by BBlack:
[operations/puppet@production] anycast healthchecker should be able to bind

https://gerrit.wikimedia.org/r/554866

gerritbot added a comment.Dec 5 2019, 3:14 PM

Change 554867 merged by BBlack:
[operations/puppet@production] anycast: bind to real service

https://gerrit.wikimedia.org/r/554867

gerritbot added a comment.Dec 5 2019, 3:14 PM

Change 554868 merged by BBlack:
[operations/puppet@production] bird: 10s delay after route withdraw

https://gerrit.wikimedia.org/r/554868

• ayounsi mentioned this in Blog Post: Internal anycast.Feb 12 2020, 3:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits