Page MenuHomePhabricator
Create Task
Maniphest T228190

Roll out Anycast RecDNS to more servers
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

Now that the Anycast RecDNS service is up and running with live traffic, we can progressively make more servers use it in their resolv.conf.

Rollout status update: things that are using anycast recdns resolv.conf in production as of 2019-07-31:

  • All hosts in edge DCs (esams, ulsfo, eqsin)
  • All cp edge cache hosts globally
  • All LVS hosts globally
  • Canary Mediawiki API and Appserver hosts in both core DCs
  • Network devices
  • Install-time stuff (as in dhcp settings and Debian installer)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
dbproxy1019: remove custom nameserver configoperations/puppetproduction+0 -3
anycast recdns: enable globallyoperations/puppetproduction+3 -148
anycast recdns: enable for codfw clientsoperations/puppetproduction+0 -1
anycast recdns: config for 41 eqiad canariesoperations/puppetproduction+123 -1
ATS: add profile::base::nameserversoperations/puppetproduction+2 -0
anycast recdns: set for canary api/appserversoperations/puppetproduction+10 -2
anycast recdns: use for all install-time DNSoperations/puppetproduction+5 -38
anycast recdns: edge sites via realm.pp (nop)operations/puppetproduction+1 -13
anycast recdns: use for all hosts at edge sitesoperations/puppetproduction+9 -0
recdns: refactor and rationalize resolv.confoperations/puppetproduction+40 -19
anycast recdns: Add to calico filtersoperations/puppetproduction+5 -0
Add anycast recdns to calico filtersoperations/deployment-chartsmaster+6 -1
anycast recdns: use for all cache_textoperations/puppetproduction+3 -6
anycast recdns: use for all cache_uploadoperations/puppetproduction+3 -4
anycast recdns: use for all LVS balancersoperations/puppetproduction+4 -25
anycast recdns: use for codfw LVS balancersoperations/puppetproduction+4 -9
anycast recdns: use for ulsfo LVS balancersoperations/puppetproduction+4 -6
anycast recdns: use for esams LVS balancersoperations/puppetproduction+4 -9
anycast recdns: use for eqsin LVS balancersoperations/puppetproduction+4 -6
Anycast recdns mon: use raw IP, for real this timeoperations/puppetproduction+10 -3
Anycast recdns monitoring: use raw IPoperations/puppetproduction+2 -2
Anycast, make recdns VIP alerts pageoperations/puppetproduction+2 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

ayounsi created this task.Jul 16 2019, 4:50 PM
ayounsi triaged this task as Medium priority.
Restricted Application added a project: SRE. · View Herald TranscriptJul 16 2019, 4:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi mentioned this in T186550: Anycast recdns.Jul 16 2019, 4:50 PM
MoritzMuehlenhoff subscribed.Jul 17 2019, 12:38 PM
BBlack moved this task from Backlog to Some old column on the Traffic board.Jul 17 2019, 3:49 PM
gerritbot added a comment.Jul 25 2019, 2:16 PM

Change 520441 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for all LVS balancers

https://gerrit.wikimedia.org/r/520441

gerritbot added a project: Patch-For-Review.Jul 25 2019, 2:16 PM

Change 525549 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for eqsin LVS balancers

https://gerrit.wikimedia.org/r/525549

gerritbot added a comment.Jul 25 2019, 2:16 PM

Change 525550 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for esams LVS balancers

https://gerrit.wikimedia.org/r/525550

gerritbot added a comment.Jul 25 2019, 2:16 PM

Change 525551 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for ulsfo LVS balancers

https://gerrit.wikimedia.org/r/525551

gerritbot added a comment.Jul 25 2019, 2:16 PM

Change 525552 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for codfw LVS balancers

https://gerrit.wikimedia.org/r/525552

gerritbot added a comment.Jul 25 2019, 2:24 PM

Change 524067 had a related patch set uploaded (by BBlack; owner: Ayounsi):
[operations/puppet@production] Anycast, make recdns VIP alerts page

https://gerrit.wikimedia.org/r/524067

gerritbot added a comment.Jul 25 2019, 2:28 PM

Change 524067 merged by BBlack:
[operations/puppet@production] Anycast, make recdns VIP alerts page

https://gerrit.wikimedia.org/r/524067

gerritbot added a comment.Jul 25 2019, 2:48 PM

Change 525556 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Anycast recdns monitoring: use raw IP

https://gerrit.wikimedia.org/r/525556

gerritbot added a comment.Jul 25 2019, 2:52 PM

Change 525556 merged by BBlack:
[operations/puppet@production] Anycast recdns monitoring: use raw IP

https://gerrit.wikimedia.org/r/525556

gerritbot added a comment.Jul 25 2019, 3:20 PM

Change 525566 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Anycast recdns mon: use raw IP, for real this time

https://gerrit.wikimedia.org/r/525566

gerritbot added a comment.Jul 25 2019, 3:28 PM

Change 525566 merged by BBlack:
[operations/puppet@production] Anycast recdns mon: use raw IP, for real this time

https://gerrit.wikimedia.org/r/525566

gerritbot added a comment.Jul 25 2019, 4:41 PM

Change 525549 merged by BBlack:
[operations/puppet@production] anycast recdns: use for eqsin LVS balancers

https://gerrit.wikimedia.org/r/525549

Stashbot added a comment.Jul 25 2019, 4:44 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T16:44:20Z] <bblack> lvs5003 - restart pybal for resolv.conf change - T228190

Stashbot added a comment.Jul 25 2019, 4:50 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T16:50:54Z] <bblack> lvs5002 - restart pybal for resolv.conf change - T228190

Stashbot added a comment.Jul 25 2019, 4:54 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T16:54:06Z] <bblack> lvs5001 - restart pybal for resolv.conf change - T228190

gerritbot added a comment.Jul 25 2019, 9:01 PM

Change 525550 merged by BBlack:
[operations/puppet@production] anycast recdns: use for esams LVS balancers

https://gerrit.wikimedia.org/r/525550

gerritbot added a comment.Jul 25 2019, 9:01 PM

Change 525551 merged by BBlack:
[operations/puppet@production] anycast recdns: use for ulsfo LVS balancers

https://gerrit.wikimedia.org/r/525551

gerritbot added a comment.Jul 25 2019, 9:01 PM

Change 525552 merged by BBlack:
[operations/puppet@production] anycast recdns: use for codfw LVS balancers

https://gerrit.wikimedia.org/r/525552

Stashbot added a comment.Jul 25 2019, 9:07 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T21:07:39Z] <bblack> backup lvses in codfw, esams, ulsfo: restart pybal for resolv.conf changes - T228190

Stashbot added a comment.Jul 25 2019, 9:38 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T21:38:20Z] <bblack> primary high-traffic1 lvses in codfw, esams, ulsfo: restart pybal for resolv.conf changes - T228190

Stashbot added a comment.Jul 25 2019, 9:47 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T21:47:05Z] <bblack> primary high-traffic2 lvses in codfw, esams, ulsfo: restart pybal for resolv.conf changes - T228190

gerritbot added a comment.Jul 25 2019, 9:57 PM

Change 520441 merged by BBlack:
[operations/puppet@production] anycast recdns: use for all LVS balancers

https://gerrit.wikimedia.org/r/520441

Stashbot added a comment.Jul 25 2019, 9:59 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T21:59:06Z] <bblack> lvs1016 - restart pybal for resolv.conf changes - T228190

Stashbot added a comment.Jul 25 2019, 10:02 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T22:02:45Z] <bblack> lvs1015 - restart pybal for resolv.conf changes - T228190

Stashbot added a comment.Jul 25 2019, 10:04 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T22:04:41Z] <bblack> lvs1014 - restart pybal for resolv.conf changes - T228190

Stashbot added a comment.Jul 25 2019, 10:07 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-25T22:07:03Z] <bblack> lvs1013 - restart pybal for resolv.conf changes - T228190

Maintenance_bot removed a project: Patch-For-Review.Jul 25 2019, 10:11 PM
BBlack added a comment.Jul 25 2019, 10:13 PM

All the LVSes are now using the anycasted recdns, which gets rid of the LVS<->recdns dependency loop and simplifies recdns server downtime processes: https://wikitech.wikimedia.org/w/index.php?title=Service_restarts&type=revision&diff=1833705&oldid=1832671

gerritbot added a comment.Jul 29 2019, 3:14 PM

Change 526163 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for all cache_upload

https://gerrit.wikimedia.org/r/526163

gerritbot added a project: Patch-For-Review.Jul 29 2019, 3:14 PM

Change 526164 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for all cache_text

https://gerrit.wikimedia.org/r/526164

gerritbot added a comment.Jul 29 2019, 3:18 PM

Change 526163 merged by BBlack:
[operations/puppet@production] anycast recdns: use for all cache_upload

https://gerrit.wikimedia.org/r/526163

gerritbot added a comment.Jul 29 2019, 3:30 PM

Change 526169 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for all hosts at edge sites

https://gerrit.wikimedia.org/r/526169

gerritbot added a comment.Jul 29 2019, 3:48 PM

Change 526164 merged by BBlack:
[operations/puppet@production] anycast recdns: use for all cache_text

https://gerrit.wikimedia.org/r/526164

gerritbot added a comment.Jul 29 2019, 4:20 PM

Change 526177 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: use for all install-time DNS

https://gerrit.wikimedia.org/r/526177

gerritbot added a comment.Jul 29 2019, 4:20 PM

Change 526178 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: Add to calico filters

https://gerrit.wikimedia.org/r/526178

Stashbot added a comment.Jul 29 2019, 10:28 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-29T22:28:41Z] <XioNoX> roll out anycast DNS and syslog to all network devices - T228190

gerritbot added a comment.Jul 30 2019, 9:07 AM

Change 526389 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Add anycast recdns to calico filters

https://gerrit.wikimedia.org/r/526389

gerritbot added a comment.Jul 30 2019, 9:18 AM

Change 526389 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Add anycast recdns to calico filters

https://gerrit.wikimedia.org/r/526389

akosiaris mentioned this in rDEPLOYCHARTS20c1649df1e1: Add anycast recdns to calico filters.Jul 30 2019, 9:18 AM
gerritbot added a comment.Jul 30 2019, 2:03 PM

Change 526178 merged by Alexandros Kosiaris:
[operations/puppet@production] anycast recdns: Add to calico filters

https://gerrit.wikimedia.org/r/526178

gerritbot added a comment.Jul 30 2019, 4:03 PM

Change 526465 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] recdns: refactor and rationalize resolv.conf

https://gerrit.wikimedia.org/r/526465

gerritbot added a comment.Jul 30 2019, 4:59 PM

Change 526465 merged by BBlack:
[operations/puppet@production] recdns: refactor and rationalize resolv.conf

https://gerrit.wikimedia.org/r/526465

gerritbot added a comment.Jul 30 2019, 5:24 PM

Change 526169 merged by BBlack:
[operations/puppet@production] anycast recdns: use for all hosts at edge sites

https://gerrit.wikimedia.org/r/526169

gerritbot added a comment.Jul 31 2019, 2:26 PM

Change 526684 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: edge sites via realm.pp (nop)

https://gerrit.wikimedia.org/r/526684

gerritbot added a comment.Jul 31 2019, 2:26 PM

Change 526685 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: set for canary api/appservers

https://gerrit.wikimedia.org/r/526685

gerritbot added a comment.Jul 31 2019, 2:42 PM

Change 526684 merged by BBlack:
[operations/puppet@production] anycast recdns: edge sites via realm.pp (nop)

https://gerrit.wikimedia.org/r/526684

gerritbot added a comment.Jul 31 2019, 5:31 PM

Change 526177 merged by BBlack:
[operations/puppet@production] anycast recdns: use for all install-time DNS

https://gerrit.wikimedia.org/r/526177

ops-monitoring-bot added a comment.Jul 31 2019, 5:39 PM

Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts:

['cp1008.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/201907311739_bblack_92388.log.

gerritbot added a comment.Jul 31 2019, 5:46 PM

Change 526685 merged by BBlack:
[operations/puppet@production] anycast recdns: set for canary api/appservers

https://gerrit.wikimedia.org/r/526685

ops-monitoring-bot added a comment.Jul 31 2019, 5:54 PM

Completed auto-reimage of hosts:

['cp1008.wikimedia.org']

Of which those FAILED:

['cp1008.wikimedia.org']
Maintenance_bot removed a project: Patch-For-Review.Jul 31 2019, 6:11 PM
BBlack added a comment.Jul 31 2019, 6:32 PM

Rollout status update: things that are using anycast recdns resolv.conf in production as of 2019-07-31:

  • All hosts in edge DCs (esams, ulsfo, eqsin)
  • All cp edge cache hosts globally
  • All LVS hosts globally
  • Canary Mediawiki API and Appserver hosts in both core DCs
  • Network devices
  • Install-time stuff (as in dhcp settings and Debian installer)
BBlack updated the task description. (Show Details)Jul 31 2019, 6:32 PM
gerritbot added a comment.Jul 31 2019, 10:27 PM

Change 526788 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: enable for codfw clients

https://gerrit.wikimedia.org/r/526788

gerritbot added a project: Patch-For-Review.Jul 31 2019, 10:27 PM
gerritbot added a comment.Aug 6 2019, 12:14 PM

Change 528440 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: add profile::base::nameservers

https://gerrit.wikimedia.org/r/528440

gerritbot added a comment.Aug 6 2019, 1:36 PM

Change 528440 merged by Ema:
[operations/puppet@production] ATS: add profile::base::nameservers

https://gerrit.wikimedia.org/r/528440

gerritbot added a comment.Aug 6 2019, 5:20 PM

Change 528524 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: config for many eqiad canaries

https://gerrit.wikimedia.org/r/528524

gerritbot added a comment.Aug 6 2019, 5:20 PM

Change 528525 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] anycast recdns: enable globally

https://gerrit.wikimedia.org/r/528525

BBlack added a comment.Aug 14 2019, 4:22 PM

I'm not sure if it goes as a subtask here, or of T167841 and/or T227808 - but recording here so we don't forget, from an earlier IRC conversation:

As things stand, if eqiad or codfw were to lose both local recursors and thus all local adverts of the anycast, they don't differentiate all the possible remote fallback options (e.g. the opposite core site vs much higher-latency alternatives like the edge sites), because we're letting the anycast advert propagate around our network freely from everywhere to everywhere. While core<->core is only a ~40ms fallback, reaching out to the far-flung edges for recursor fallback could raise recdns latency for core-site services out into the 200ms+ range, which would probably have a higher chance of negative impact. There's also just a lot more services and code in the core sites (vs the small stack of stuff in the edges), and there will probably always be some that use recdns in less-than-ideal ways and thus are very sensitive to it.

What we talked about on IRC was changing the router filtering for the anycast stuff such that the core sites successfully advertise the internal anycast space globally (are fallbacks for the opposite core site and all edges), but the edge sites do not advertise the anycast space back towards the core or other edge sites (so that if they lose their local recursors, they fall back to one of the core sites). We should probably block on this before doing any whole-site conversion of either eqiad or codfw to anycast recdns, just in case!

BBlack added a parent task: T186550: Anycast recdns.Aug 15 2019, 2:26 PM
BBlack mentioned this in T98006: Anycast AuthDNS.Aug 15 2019, 2:48 PM
ayounsi mentioned this in T227808: Standardize cross confederation BGP policies.Aug 15 2019, 10:42 PM
Stashbot added a comment.Aug 20 2019, 5:57 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-20T17:57:55Z] <bblack> deploying anycast recdns settings to resolv.conf on 41 live hosts in eqiad - https://gerrit.wikimedia.org/r/528524 - T228190

gerritbot added a comment.Aug 20 2019, 5:58 PM

Change 528524 merged by BBlack:
[operations/puppet@production] anycast recdns: config for 41 eqiad canaries

https://gerrit.wikimedia.org/r/528524

ayounsi closed subtask T230955: Configure Layer3 hashing for router ECMP (for anycast DNS) as Resolved.Aug 22 2019, 12:27 AM
gerritbot added a comment.Aug 27 2019, 2:48 PM

Change 526788 merged by BBlack:
[operations/puppet@production] anycast recdns: enable for codfw clients

https://gerrit.wikimedia.org/r/526788

Stashbot added a comment.Aug 27 2019, 2:49 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-27T14:48:59Z] <bblack> deploying anycast recdns resolv.conf setting to all codfw - T228190

gerritbot added a comment.Sep 17 2019, 2:18 PM

Change 528525 merged by BBlack:
[operations/puppet@production] anycast recdns: enable globally

https://gerrit.wikimedia.org/r/528525

gerritbot added a comment.Sep 17 2019, 2:34 PM

Change 537448 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] dbproxy1019: remove custom nameserver config

https://gerrit.wikimedia.org/r/537448

gerritbot added a comment.Sep 17 2019, 2:37 PM

Change 537448 merged by BBlack:
[operations/puppet@production] dbproxy1019: remove custom nameserver config

https://gerrit.wikimedia.org/r/537448

Maintenance_bot removed a project: Patch-For-Review.Sep 17 2019, 3:11 PM
faidon mentioned this in rOHPUe26faf3e6bdb: Use anycast DNS IP.Oct 3 2019, 11:11 AM
BBlack closed this task as Resolved.Nov 5 2019, 5:56 PM
BBlack assigned this task to ayounsi.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits