Page MenuHomePhabricator
Create Task
Maniphest T342621

eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP
Closed, ResolvedPublic
Actions

Description

In the legacy network model, each designate DNS recoursor would have its own public IPv4 address. In the new model, there is a single BGP anycast VIP shared between all the recursors. See also: T307357: Move cloud vps ns-recursor IPs to host/row-independent addressing

So as part of T341060: openstack eqiad1: introduce cloud-private and cloudlb we need to transition
from:

  • ns-recursor0.openstack.eqiad1.wikimediacloud.org
  • ns-recursor1.openstack.eqiad1.wikimediacloud.org

to:

  • ns-recursor.openstack.eqiad1.wikimediacloud.org

Note, we are using ns-recursor-next.openstack.eqiad1.wikimediacloud.org as placeholder while preparing the code changes, etc.

Details

SubjectRepoBranchLines +/-
wmcs: remove ns-recursorX FQDNsoperations/puppetproduction+5 -20
cr-cloud: add ns-recursor.openstack.eqiad1operations/homer/publicmaster+3 -1
wmcs: drop cloudservices1004 addressesoperations/dnsmaster+0 -4
wmcs: refresh DNS addressesoperations/puppetproduction+7 -8
openstack: refresh cloudservices1006 ns addressoperations/puppetproduction+6 -6
openstack: eqiad1: drop -next suffix from ns-recursoroperations/puppetproduction+3 -4
cloudservices1006: prepare serviceoperations/puppetproduction+39 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedaborreroT296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet
 ResolvedaborreroT297596 have cloud hardware servers in the cloud realm using a dedicated LB layer
 ResolvedaborreroT324992 cloudlb: create PoC on codfw
 ResolvedaborreroT327908 rename cloudgw2001-dev into cloudlb2001-dev
 ResolvedayounsiT327930 Is Vlan 2122 cloud-support1-b-codfw required?
 ResolvedcmooneyT327919 Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it
 ResolvedPapaulT331470 Run 2x1G links from asw-b1-codfw to cloudsw1-b1-codfw
Restricted Task
 ResolvedcmooneyT333316 Homer unable to commit config to cloudsw1-b1-codfw (QFX5120 21.4R3.16)
 ResolvedcmooneyT336368 cloudgw: review security policy for edge network
 ResolvedaborreroT332153 cloudlb PoC: prepare backends
 ResolvedaborreroT336963 cloudcontrol2001-dev can't reach cloud-vps public IPs
 ResolvedaborreroT335759 cloud-private subnet: introduce new domain
 ResolvedtaaviT336566 cloud: failures resolving some `wikimedia.cloud` domains
 OpencmooneyT346428 Netbox: Add support for our complex host network setups in provision script
 ResolvedaborreroT346410 need private IPs for cloudvirt200[4-6]-dev.codfw.wmnet
 OpencmooneyT347375 Netbox device location information not available on the first Puppet run of a device
 ResolvedaborreroT335760 Modify Bird module to allow source IP to be passed to template
 ResolvedaborreroT336071 cloudlb: figure out routing
 ResolvedaborreroT337758 puppet: interface::route resource does not persists settings
 ResolvedaborreroT338936 cloudlb: figure out plans for eqiad1
 ResolvedaborreroT338937 cloudlb: review swift/radosgw status
ResolvedAndrewT341380 Open swift port (28080) to the public internet
 ResolvedAndrewT341484 Horizon object store UI doesn't allow uploading of files
 ResolvedAndrewT341640 Move Horizon deploy to a Docker container
 OpenAndrewT341509 radosgw+keystone chokes on projects with '-' in their id
 OpenAndrewT343158 Support OpenStack projects where project name != project id
 ResolvedAndrewT347927 Better swift error messages for projects with invalid chars
 ResolvedaborreroT340536 cloud-private: figure out, implement and test cross-DC traffic
 ResolvedtaaviT341060 openstack eqiad1: introduce cloud-private and cloudlb
 OpenNoneT317177 [tracking] Don't keep on the public vlans hosts that don't require it
 OpenNoneT340446 Cloud VPS Designate setup improvements
 ResolvedaborreroT307357 Move cloud vps ns-recursor IPs to host/row-independent addressing
 ResolvedcmooneyT336587 cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan
 ResolvedaborreroT342621 eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP
 InvalidaborreroT346092 cloudcontrols can't reach ns-recursor.openstack.eqiad1.wikimediacloud.org
 OpenaborreroT346426 Some VPS instances still using ns-recursor0

Event Timeline

aborrero created this task.Jul 25 2023, 11:26 AM
gerritbot added a comment.Jul 25 2023, 11:27 AM

Change 941383 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudservices1006: prepare service

https://gerrit.wikimedia.org/r/941383

gerritbot added a project: Patch-For-Review.Jul 25 2023, 11:27 AM
aborrero mentioned this in T341060: openstack eqiad1: introduce cloud-private and cloudlb.Jul 25 2023, 11:27 AM
aborrero triaged this task as Medium priority.Jul 25 2023, 11:35 AM
aborrero added a parent task: T307357: Move cloud vps ns-recursor IPs to host/row-independent addressing.
aborrero moved this task from Backlog to Next on the User-aborrero board.
fnegri edited projects, added cloud-services-team (FY2023/2024-Q1-Q2); removed cloud-services-team (FY2022/2023-Q4).Aug 7 2023, 2:37 PM
aborrero changed the task status from Open to Stalled.Aug 29 2023, 12:39 PM
aborrero moved this task from Next to Blocked on the User-aborrero board.

blocked on T342161: Q1:rack/setup/install cloudservices1006.eqiad.wmnet we need the server to be available.

aborrero changed the task status from Stalled to In Progress.Aug 30 2023, 9:57 AM
aborrero moved this task from Blocked to Doing on the User-aborrero board.
aborrero mentioned this in T345240: cloudservices1006: put into service.Aug 30 2023, 10:45 AM
gerritbot added a comment.Aug 30 2023, 10:48 AM

Change 941383 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudservices1006: prepare service

https://gerrit.wikimedia.org/r/941383

Maintenance_bot removed a project: Patch-For-Review.Aug 30 2023, 11:11 AM
gerritbot added a comment.Sep 4 2023, 9:51 AM

Change 954605 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: eqiad1: drop -next suffix from ns-recursor

https://gerrit.wikimedia.org/r/954605

gerritbot added a project: Patch-For-Review.Sep 4 2023, 9:51 AM
gerritbot added a comment.Sep 4 2023, 10:03 AM

Change 954605 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: eqiad1: drop -next suffix from ns-recursor

https://gerrit.wikimedia.org/r/954605

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2023, 10:11 AM
Stashbot added a comment.Sep 5 2023, 12:46 PM

Mentioned in SAL (#wikimedia-cloud) [2023-09-05T12:45:57Z] <arturo> moved all VMs to ns-recursor.openstack.eqiad1.wikimediacloud.org via project puppet (T345240, T342621)

Stashbot added a comment.Sep 6 2023, 8:47 AM

Mentioned in SAL (#wikimedia-cloud) [2023-09-06T08:47:02Z] <arturo> switch project to new DNS recursor via horizon project hiera (T345240, T342621)

aborrero moved this task from Doing to Next on the User-aborrero board.Sep 6 2023, 8:48 AM
gerritbot added a comment.Sep 11 2023, 12:08 PM

Change 956415 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wmcs: drop cloudservices1004 addresses

https://gerrit.wikimedia.org/r/956415

gerritbot added a project: Patch-For-Review.Sep 11 2023, 12:08 PM
gerritbot added a comment.Sep 11 2023, 12:14 PM

Change 956417 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: refresh cloudservices1006 ns address

https://gerrit.wikimedia.org/r/956417

gerritbot added a comment.Sep 11 2023, 12:26 PM

Change 956419 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] wmcs: refresh DNS addresses

https://gerrit.wikimedia.org/r/956419

gerritbot added a comment.Sep 11 2023, 12:32 PM

Change 956417 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: refresh cloudservices1006 ns address

https://gerrit.wikimedia.org/r/956417

gerritbot added a comment.Sep 11 2023, 12:32 PM

Change 956419 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] wmcs: refresh DNS addresses

https://gerrit.wikimedia.org/r/956419

gerritbot added a comment.Sep 11 2023, 12:33 PM

Change 956415 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wmcs: drop cloudservices1004 addresses

https://gerrit.wikimedia.org/r/956415

Stashbot added a comment.Sep 11 2023, 12:36 PM

Mentioned in SAL (#wikimedia-cloud) [2023-09-11T12:36:34Z] <arturo> update DNS resolver cloud-wide to use 172.20.255.1 (T342621)

gerritbot added a comment.Sep 11 2023, 12:52 PM

Change 956429 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] cr-cloud: add ns-recursor.openstack.eqiad1

https://gerrit.wikimedia.org/r/956429

gerritbot added a comment.Sep 11 2023, 12:53 PM

Change 956429 merged by Arturo Borrero Gonzalez:

[operations/homer/public@master] cr-cloud: add ns-recursor.openstack.eqiad1

https://gerrit.wikimedia.org/r/956429

Maintenance_bot removed a project: Patch-For-Review.Sep 11 2023, 1:10 PM
aborrero closed this task as Resolved.Sep 11 2023, 3:25 PM
gerritbot added a comment.Sep 11 2023, 3:55 PM

Change 956463 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] wmcs: remove ns-recursorX FQDNs

https://gerrit.wikimedia.org/r/956463

gerritbot added a project: Patch-For-Review.Sep 11 2023, 3:55 PM
gerritbot added a comment.Sep 11 2023, 3:57 PM

Change 956463 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] wmcs: remove ns-recursorX FQDNs

https://gerrit.wikimedia.org/r/956463

Maintenance_bot removed a project: Patch-For-Review.Sep 11 2023, 4:10 PM
LSobanski mentioned this in T346060: GitLab Runners in WMCS are offline.Sep 11 2023, 4:17 PM
dancy subscribed.Sep 13 2023, 3:09 PM

Puppet is failing to run on a bunch of integration project nodes. For example, on integration-agent-docker-1052.integration.eqiad1.wikimedia.cloud:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Method call, DNS lookup failed for ns-recursor1.openstack.eqiad1.wikimediacloud.org Resolv::DNS::Resource::IN::A (file: /etc/puppet/modules/resolvconf/manifests/init.pp, line: 25, column: 34) on node integration-agent-docker-1052.integration.eqiad1.wikimedia.cloud

This seems related to this ticket.

taavi subscribed.Sep 13 2023, 3:32 PM
In T342621#9164192, @dancy wrote:

Puppet is failing to run on a bunch of integration project nodes. For example, on integration-agent-docker-1052.integration.eqiad1.wikimedia.cloud:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Method call, DNS lookup failed for ns-recursor1.openstack.eqiad1.wikimediacloud.org Resolv::DNS::Resource::IN::A (file: /etc/puppet/modules/resolvconf/manifests/init.pp, line: 25, column: 34) on node integration-agent-docker-1052.integration.eqiad1.wikimedia.cloud

This seems related to this ticket.

integration-puppetmaster-02 has local cherry-picks that have been blocking the git-sync-upstream process since early July. Fixing Puppet git repository updates will resolve the Puppet failures you're seeing.

dancy added a comment.Sep 13 2023, 4:04 PM

Thank you @taavi. I'll take a look.

dancy added a comment.Sep 13 2023, 5:24 PM

I removed the offending commit from integration-puppetmaster-02.integration.eqiad.wmflabs:/var/lib/git/operations/puppet. Thanks again @taavi for the pointer to the root problem.

aborrero closed subtask T346092: cloudcontrols can't reach ns-recursor.openstack.eqiad1.wikimediacloud.org as Invalid.Sep 15 2023, 8:53 AM
aborrero added a subtask: T346426: Some VPS instances still using ns-recursor0.Sep 15 2023, 9:29 AM
aborrero closed subtask T346426: Some VPS instances still using ns-recursor0 as Resolved.Sep 15 2023, 12:57 PM
Don-vip mentioned this in T346471: DNS name resolution failure with www.spacecom.mil from Cloud VPS.Sep 16 2023, 12:06 PM
fnegri moved this task from Backlog to Done on the cloud-services-team (FY2023/2024-Q1-Q2) board.Sep 20 2023, 9:10 AM
cmooney reopened subtask T346426: Some VPS instances still using ns-recursor0 as Open.Feb 8 2024, 6:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL