Page MenuHomePhabricator
Create Task
Maniphest T307357

Move cloud vps ns-recursor IPs to host/row-independent addressing
Closed, ResolvedPublic
Actions

Description

Currently Cloud VPS DNS recursors use simple reserved VIPs (one extra VIP per cloudservices* box) from the row-specific public1 subnets. This has some downsides:

  • Requiring that cloudservices* hosts have public addresses in a specific physical DC row
  • No failover when rebooting hosts
    • This point is especially problematic since surprisingly{{cn}} many DNS implementations will misbehave when the first listed recursor is down

This task is here to investigate if we can replace the current system with something (LVS, Anycast, ...) that addresses those problems. Note that the same hosts also host the auth dns for WMCS domains using their primary (public) IP addresses.

see also: T207533: Move labs-recursors in WMCS T119660: Set up LVS for labs dns recursors

Current work is to implement this layout:

cloudservices2004-dev

Interface IPs, same as every other cloud host:

InterfaceIPVlanDomain
eno110.192.20.26/24cloud-hosts1-codfwcloudservices2004-dev.codfw.wmnet
vlan2151@eno1172.20.5.8/24cloud-private-b1-codfwcloudservices2004-dev.private.codfw.wikimedia.cloud

BGP IPs announced to 172.20.5.1 over vlan2151:

IPDNSDescription
185.15.57.25ns0.openstack.codfw1dev.wikimediacloud.orgpdns_server listens on this, pdns_recursor uses it for outbound queries
172.20.254.1ns-recursor.openstack.codfw1dev.wikimediacloud.orgpdns_recursor listens on this for client queries

cloudservices2005-dev

Interface IPs, same as every other cloud host:

InterfaceIPVlanDomain
eno110.192.20.27/24cloud-hosts1-codfwcloudservices2005-dev.codfw.wmnet
vlan2151@eno1172.20.5.9/24cloud-private-b1-codfwcloudservices2005-dev.private.codfw.wikimedia.cloud

BGP IPs announced to 172.20.5.1 over vlan2151:

IPDNSDescription
185.15.57.26ns1.openstack.codfw1dev.wikimediacloud.orgpdns_server listens on this, pdns_recursor uses it for outbound queries
172.20.254.1ns-recursor.openstack.codfw1dev.wikimediacloud.orgpdns_recursor listens on this for client queries

Details

SubjectRepoBranchLines +/-
wikimediacloud.org: decom ns-recursor0.openstack.eqiad1.wikimediacloud.orgoperations/dnsmaster+0 -5
cr-cloud: Remove unused termsoperations/homer/publicmaster+0 -52
reports/network: ignore IPv6 for cloudservices boxesoperations/software/netbox-extrasmaster+1 -0
wikimediacloud.org: refresh A for ns1.openstack.codfw1dev.wikimediacloud.orgoperations/dnsmaster+1 -1
wikimediacloud.org: cleanup for ns.openstack.codfw1dev.wikimediacloud.orgoperations/dnsmaster+2 -3
cloudservices2005-dev: fix DNS addressoperations/puppetproduction+2 -3
dnsrecursor: introduce query-local-address parameteroperations/puppetproduction+7 -0
openstack: codfw1dev: more hiera adjustementsoperations/puppetproduction+13 -4
openstack: codfw1dev: designate: fix recursor_service_nameoperations/puppetproduction+2 -4
openstack: codfw1dev: make designate use cloud-private address by defaultoperations/puppetproduction+1 -19
openstack: pdns: auth: service: drop IPv6 supportoperations/puppetproduction+2 -3
wikimediacloud.org: eqiad1: drop IPv6 records for DNS servicesoperations/dnsmaster+0 -6
wikimediacloud.org: drop unused ns-recursor[0,1] FQDNsoperations/dnsmaster+0 -4
57.15.185.in-addr.arpa: fix delegationoperations/dnsmaster+4 -2
wikimedia.cloud: fix delegation for codfw1devoperations/dnsmaster+2 -1
wikimediacloud.org: refresh ns0.openstack.codfw1dev addressoperations/dnsmaster+2 -3
wikimediacloud.org: codfw1dev: drop IPv6 records for DNS servicesoperations/dnsmaster+0 -4
acme_chief: openstack: codfw1dev: refresh LDAP certificatesoperations/puppetproduction+4 -9
acme_chief: openstack: codfw1dev: allow cloudservices2004-devoperations/puppetproduction+10 -4
openstack: pdns: recursor: make it listen in the right addressoperations/puppetproduction+1 -1
openstack: pdns: recursor: drop IPv6 supportoperations/puppetproduction+10 -13
openstack: codfw1dev: pdns: use modern auth server for forward zonesoperations/puppetproduction+6 -18
cloudservices2004-dev: put into service with new setupoperations/puppetproduction+16 -6
openstack: codfw1dev: services: add support for BGP public IPv4 addressesoperations/puppetproduction+14 -6
wikimediacloud.org: refresh FQDNs for codfw1dev DNS serversoperations/dnsmaster+7 -15
Add private cloud-vips range to prefix list for cloudswoperations/homer/publicmaster+1 -0
openstack: designate: auth: allow configuring ips to listen onoperations/puppetproduction+10 -1
openstack: designate: recursor: add new BGP VIPoperations/puppetproduction+10 -2
cloudservices: codfw1dev: refresh VIPsoperations/puppetproduction+10 -9
cloudservices: codfw1dev: enable cloud-private subnetoperations/puppetproduction+16 -0
wikimedia.cloud: add cloudservices200[4/5]-dev cloud-private addressoperations/dnsmaster+4 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedPapaulT342076 Decommission asw-b1-codfw
 ResolvedaborreroT296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet
 ResolvedaborreroT297596 have cloud hardware servers in the cloud realm using a dedicated LB layer
 ResolvedaborreroT324992 cloudlb: create PoC on codfw
 ResolvedaborreroT327908 rename cloudgw2001-dev into cloudlb2001-dev
 ResolvedayounsiT327930 Is Vlan 2122 cloud-support1-b-codfw required?
 ResolvedcmooneyT327919 Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it
 ResolvedPapaulT331470 Run 2x1G links from asw-b1-codfw to cloudsw1-b1-codfw
Restricted Task
 ResolvedcmooneyT333316 Homer unable to commit config to cloudsw1-b1-codfw (QFX5120 21.4R3.16)
 ResolvedcmooneyT336368 cloudgw: review security policy for edge network
 ResolvedaborreroT332153 cloudlb PoC: prepare backends
 ResolvedaborreroT336963 cloudcontrol2001-dev can't reach cloud-vps public IPs
 ResolvedaborreroT335759 cloud-private subnet: introduce new domain
 ResolvedtaaviT336566 cloud: failures resolving some `wikimedia.cloud` domains
 OpencmooneyT346428 Netbox: Add support for our complex host network setups in provision script
 ResolvedaborreroT346410 need private IPs for cloudvirt200[4-6]-dev.codfw.wmnet
 OpencmooneyT347375 Netbox device location information not available on the first Puppet run of a device
 ResolvedaborreroT335760 Modify Bird module to allow source IP to be passed to template
 ResolvedaborreroT336071 cloudlb: figure out routing
 ResolvedaborreroT337758 puppet: interface::route resource does not persists settings
 ResolvedaborreroT338936 cloudlb: figure out plans for eqiad1
 ResolvedaborreroT338937 cloudlb: review swift/radosgw status
ResolvedAndrewT341380 Open swift port (28080) to the public internet
 ResolvedAndrewT341484 Horizon object store UI doesn't allow uploading of files
 ResolvedAndrewT341640 Move Horizon deploy to a Docker container
 OpenAndrewT341509 radosgw+keystone chokes on projects with '-' in their id
 OpenAndrewT343158 Support OpenStack projects where project name != project id
 ResolvedAndrewT347927 Better swift error messages for projects with invalid chars
 ResolvedaborreroT340536 cloud-private: figure out, implement and test cross-DC traffic
 OpenNoneT317177 [tracking] Don't keep on the public vlans hosts that don't require it
 OpenNoneT340446 Cloud VPS Designate setup improvements
 ResolvedaborreroT307357 Move cloud vps ns-recursor IPs to host/row-independent addressing
 ResolvedcmooneyT336587 cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan
 ResolvedaborreroT338433 openstack: figure out how to change /etc/resolv.conf in all VMs
 ResolvedaborreroT338778 cloudservices2004-dev: reimage into new network setup
 InvalidNoneT339869 codfw1dev: LDAP database content seems to have lost years of content
 ResolvedaborreroT338779 cloudservices2005-dev: reimage into new network setup
 ResolvedaborreroT340047 LDAP problems following cloudservices2005-dev reimage
 OpenNoneT340127 systemctl restart keystone doesn't actually restart keystone sometimes
 ResolvedaborreroT340488 codfw1dev: OpenStack services can only sort of talk to memacached on cloudcontrols
 InvalidaborreroT338938 codfw1dev: finish auth DNS server transition
 ResolvedaborreroT339905 codfw1dev: LDAP setup needs a refresh so TLS works
 ResolvedfgiunchediT341966 prometheus in codfw can't reach cloudservices' pdns to fetch metrics
 ResolvedaborreroT342621 eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP
 InvalidaborreroT346092 cloudcontrols can't reach ns-recursor.openstack.eqiad1.wikimediacloud.org
 OpenaborreroT346426 Some VPS instances still using ns-recursor0
 ResolvedcmooneyT347858 Manage WMCS codfw public service VIP DNS from Netbox

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
aborrero moved this task from Doing to Radar on the User-aborrero board.Jun 12 2023, 3:18 PM
aborrero mentioned this in T338929: openstack designate: introduce some as-code mechanism to instrument DNS data.Jun 13 2023, 9:37 AM
aborrero added a subtask: T338929: openstack designate: introduce some as-code mechanism to instrument DNS data.Jun 13 2023, 9:41 AM
aborrero mentioned this in T338938: codfw1dev: finish auth DNS server transition.Jun 13 2023, 10:45 AM
aborrero removed a subtask: T338929: openstack designate: introduce some as-code mechanism to instrument DNS data.
taavi added a comment.Jun 13 2023, 4:23 PM

The current implementation on codfw1dev seems to have forgotten that the recursors need outbound access to the public internet to perform recursive queries. Right now the only public address on the DNS servers is the ns.openstack.codfw1dev.wikimediacloud.org VIP, which can't be used for this purpose as queries sent via it won't be routed consistently back (as they might be routed to the other anycast backends).

aborrero changed the task status from Stalled to Open.Jun 13 2023, 4:32 PM
aborrero moved this task from Radar to Doing on the User-aborrero board.
cmooney added a comment.Jun 13 2023, 4:42 PM
In T307357#8928378, @taavi wrote:

The current implementation on codfw1dev seems to have forgotten that the recursors need outbound access to the public internet to perform recursive queries. Right now the only public address on the DNS servers is the ns.openstack.codfw1dev.wikimediacloud.org VIP, which can't be used for this purpose as queries sent via it won't be routed consistently back (as they might be routed to the other anycast backends).

That's a very good point yeah. The recursors will need their own unique public IPs to source external queries from. I'd thought about this before but somehow didn't cop during the most recent discussion when we opted for anycast.

@aborrero we can assign a new, unique public IP to each. If it simplifies things we can maybe discard the Anycast IP for the recursor service, and configure clients with multiple IPs (each belonging to a separate cloudservice node). That may make the dns service easier to config, as you tell it to only listen on a single IP, which it also uses to make outbound queries.

Alternately we can keep the private anycast IP in play, and clients can continue to send queries to that. As long as the recursor can be configured to use its unique public IP for outbound connections that will work.

cmooney added a comment.Jun 13 2023, 4:55 PM

@aborrero I discussed the idea of a dedicated cloud-public vlan on the wikitech page previously.

TL;DR its not ideal as it burns a load of public IP addresses. But we may need to do it if there is no way to tell designate "this is the IP to use for outbound requests".

aborrero added a comment.Jun 14 2023, 9:22 AM
In T307357#8928451, @cmooney wrote:

@aborrero I discussed the idea of a dedicated cloud-public vlan on the wikitech page previously.

TL;DR its not ideal as it burns a load of public IP addresses. But we may need to do it if there is no way to tell designate "this is the IP to use for outbound requests".

I think that's the query-local-addressoption. Upstream docs:

Similar to this https://gerrit.wikimedia.org/r/c/operations/puppet/+/929739/ patch merged yesterday which I believe was what triggered @taavi into raising this problem.

Could you describe the setup you have in mind? Would it be a static public IPv4 address /32 per server plus a route in the cloudsw route?
If so, what would be the benefit of the BGP thingy?

cmooney added a comment.EditedJun 14 2023, 9:28 AM
In T307357#8930600, @aborrero wrote:

I think that's the query-local-addressoption. Upstream docs:

That's it yep.

Similar to this https://gerrit.wikimedia.org/r/c/operations/puppet/+/929739/ patch merged yesterday which I believe was what triggered @taavi into raising this problem.

Seems like that is how we can set it from puppet yep.

Could you describe the setup you have in mind? Would it be a static public IPv4 address /32 per server plus a route in the cloudsw route?

We assign a new unique /32 IP address to each cloudservice node in Netbox, and have each of them announce this to the switch with BGP, in addition to the two VIPs they are already announcing.

If so, what would be the benefit of the BGP thingy?

It saves IP addresses as discussed. For instance in eqiad assuming we put the two servers in different racks we need to use at least 16 IP addresses to create 2 cloud-private subnets. With this approach we only use 2 IPs not 16, and it fits neatly into the setup we use for Anycast VIPs.

aborrero added a comment.EditedJun 14 2023, 9:46 AM
In T307357#8930604, @cmooney wrote:

Could you describe the setup you have in mind? Would it be a static public IPv4 address /32 per server plus a route in the cloudsw route?

We assign a new unique /32 IP address to each cloudservice node in Netbox, and have each of them announce this to the switch with BGP, in addition to the two VIPs they are already announcing.

If so, what would be the benefit of the BGP thingy?

It saves IP addresses as discussed. For instance in eqiad assuming we put the two servers in different racks we need to use at least 16 IP addresses to create 2 cloud-private subnets. With this approach we only use 2 IPs not 16, and it fits neatly into the setup we use for Anycast VIPs.

This is what I'm understanding of your proposal:

  • cloudservices2004-dev.codfw.wmnet <-- for SSH, puppet, etc
    • cloudservices2004-dev.private.codfw.wikimedia.cloud <-- address in cloud-private subnet for comms with the rest of the openstack services
    • query-local-address.cloudservices2004-dev.openstack.codfw1dev.wikimediacloud.org <-- public IPv4 address used for outbound recursive queries. BGP announced (but not shared).
    • ns.openstack.codfw1dev.wikimediacloud.org <-- public IPv4 address used for auth DNS, BGP announced and shared with the other node.
    • ns-recursor.openstack.codfw1dev.wikimediacloud.org <-- priv IPv4 address used for recursor DNS, BGP announced and shared with the other node.
  • cloudservices2005-dev.codfw.wmnet <-- for SSH, puppet, etc
    • cloudservices2005-dev.private.codfw.wikimedia.cloud <-- address in cloud-private subnet for comms with the rest of the openstack services
    • query-local-address.cloudservices2005-dev.openstack.codfw1dev.wikimediacloud.org <-- public IPv4 address used for outbound recursive queries. BGP announced (but not shared).
    • ns.openstack.codfw1dev.wikimediacloud.org <-- public IPv4 address used for auth DNS, BGP announced and shared with the other node.
    • ns-recursor.openstack.codfw1dev.wikimediacloud.org <-- priv IPv4 address used for recursor DNS, BGP announced and shared with the other node.

My point is that we could go with the 2 public IPv4 addresses for both recursive and authoritative and that would be a way simpler setup.

I think I'm proposing this:

  • cloudservices2004-dev.codfw.wmnet <-- for SSH, puppet, etc
    • cloudservices2004-dev.private.codfw.wikimedia.cloud <-- address in cloud-private subnet for comms with the rest of the openstack services
    • ns0.openstack.codfw1dev.wikimediacloud.org <-- public IPv4 address used for both auth and recursor. BGP announced (but not shared) to don't waste IPv4 and to make it row-independent
  • cloudservices2005-dev.codfw.wmnet <-- for SSH, puppet, etc
    • cloudservices2005-dev.private.codfw.wikimedia.cloud <-- address in cloud-private subnet for comms with the rest of the openstack services
    • ns1.openstack.codfw1dev.wikimediacloud.org <-- public IPv4 address used for both auth and recursor. BGP announced (but not shared) to don't waste IPv4 and to make it row-independent

Note the single BGP address (not shared) per server works for everything, auth and recursor DNS.

aborrero added a comment.EditedJun 14 2023, 11:11 AM
In T307357#8930653, @aborrero wrote:

I think I'm proposing this:

Talking with @taavi on IRC, he pointed that udp 53 can't be used for both auth and recursor without significant rework of the PDNS setup we have.

So, adding the shared address for the recursor it would be:

  • cloudservices2004-dev.codfw.wmnet <-- for SSH, puppet, etc
    • cloudservices2004-dev.private.codfw.wikimedia.cloud <-- address in cloud-private subnet for comms with the rest of the openstack services
    • ns-recursor.openstack.codfw1dev.wikimediacloud.org <-- priv IPv4 address used for recursor DNS, BGP announced and shared with the other node. (existing)
    • ns0.openstack.codfw1dev.wikimediacloud.org <-- public IPv4 address used for both auth and recursor query-local-address. BGP announced (but not shared) to don't waste IPv4 and to make it row-independent (net new)
  • cloudservices2005-dev.codfw.wmnet <-- for SSH, puppet, etc
    • cloudservices2005-dev.private.codfw.wikimedia.cloud <-- address in cloud-private subnet for comms with the rest of the openstack services
    • ns-recursor.openstack.codfw1dev.wikimediacloud.org <-- priv IPv4 address used for recursor DNS, BGP announced and shared with the other node. (existing)
    • ns1.openstack.codfw1dev.wikimediacloud.org <-- public IPv4 address used for both auth and recursor query-local-address. BGP announced (but not shared) to don't waste IPv4 and to make it row-independent (net new)
cmooney added a comment.EditedJun 14 2023, 11:11 AM
In T307357#8930653, @aborrero wrote:

My point is that we could go with the 2 public IPv4 addresses for both recursive and authoritative and that would be a way simpler setup.

Sure. That was my original proposal on the enhancement page, but I now realise it can't work quite like that. The problem is the pdns_server and pdns_recursor cannot both bind to <public_ip>:53. You need to have two separate IPs for the two daemons to co-exist listening on the same port.

The best way to address that constraint, I think, is to keep the 172.20.254.1/32 Anycast VIP as the IP the recursor listens on.

We can move away from the Anycast setup for the auth dns service though, assigning a unique IP per server, and moving back to having the NS1/NS2 entries in the parent zone.

There is no problem re-using this IP on each server for the recursor to use outbound with query-local-address.

Example

So what you ned up with looks something like this.

cloudservices2004-dev

Interface IPs, same as every other cloud host:

InterfaceIPVlanDomain
eno110.192.20.26/24cloud-hosts1-codfwcloudservices2004-dev.codfw.wmnet
vlan2151@eno1172.20.5.8/24cloud-private-b1-codfwcloudservices2004-dev.private.codfw.wikimedia.cloud

BGP IPs announced to 172.20.5.1 over vlan2151:

IPDNSDescription
185.15.57.25ns0.openstack.codfw1dev.wikimediacloud.orgpdns_server listens on this, pdns_recursor uses it for outbound queries
172.20.254.1ns-recursor.openstack.codfw1dev.wikimediacloud.orgpdns_recursor listens on this for client queries

cloudservices2005-dev

Interface IPs, same as every other cloud host:

InterfaceIPVlanDomain
eno110.192.20.27/24cloud-hosts1-codfwcloudservices2005-dev.codfw.wmnet
vlan2151@eno1172.20.5.9/24cloud-private-b1-codfwcloudservices2005-dev.private.codfw.wikimedia.cloud

BGP IPs announced to 172.20.5.1 over vlan2151:

IPDNSDescription
185.15.57.26ns1.openstack.codfw1dev.wikimediacloud.orgpdns_server listens on this, pdns_recursor uses it for outbound queries
172.20.254.1ns-recursor.openstack.codfw1dev.wikimediacloud.orgpdns_recursor listens on this for client queries
aborrero added a comment.Jun 14 2023, 11:13 AM

Ok, I think we are in the same page!

gerritbot added a comment.Jun 14 2023, 1:01 PM

Change 930170 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: services: add support for BGP public IPv4 addresses

https://gerrit.wikimedia.org/r/930170

gerritbot added a project: Patch-For-Review.Jun 14 2023, 1:01 PM
gerritbot added a comment.Jun 14 2023, 4:07 PM

Change 930170 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: services: add support for BGP public IPv4 addresses

https://gerrit.wikimedia.org/r/930170

Maintenance_bot removed a project: Patch-For-Review.Jun 14 2023, 4:10 PM
gerritbot added a comment.Jun 14 2023, 4:16 PM

Change 930210 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudservices2005-dev: fix DNS address

https://gerrit.wikimedia.org/r/930210

gerritbot added a project: Patch-For-Review.Jun 14 2023, 4:16 PM
gerritbot added a comment.Jun 14 2023, 4:36 PM

Change 930212 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudservices2004-dev: put into service with new setup

https://gerrit.wikimedia.org/r/930212

gerritbot added a comment.Jun 15 2023, 9:24 AM

Change 930212 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudservices2004-dev: put into service with new setup

https://gerrit.wikimedia.org/r/930212

gerritbot added a comment.Jun 15 2023, 10:19 AM

Change 930556 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: pdns: use modern auth server for forward zones

https://gerrit.wikimedia.org/r/930556

gerritbot added a comment.Jun 15 2023, 11:50 AM

Change 930556 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: pdns: use modern auth server for forward zones

https://gerrit.wikimedia.org/r/930556

gerritbot added a comment.Jun 15 2023, 12:33 PM

Change 930616 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: pdns: recursor: drop IPv6 support

https://gerrit.wikimedia.org/r/930616

gerritbot added a comment.Jun 15 2023, 2:25 PM

Change 930616 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: pdns: recursor: drop IPv6 support

https://gerrit.wikimedia.org/r/930616

gerritbot added a comment.Jun 15 2023, 2:40 PM

Change 930642 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: pdns: recursor: make it listen in the right address

https://gerrit.wikimedia.org/r/930642

gerritbot added a comment.Jun 15 2023, 2:44 PM

Change 930642 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: pdns: recursor: make it listen in the right address

https://gerrit.wikimedia.org/r/930642

gerritbot added a comment.Jun 15 2023, 3:00 PM

Change 930647 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: codfw1dev: drop IPv6 records for DNS services

https://gerrit.wikimedia.org/r/930647

gerritbot added a comment.Jun 15 2023, 3:00 PM

Change 930648 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: eqiad1: drop IPv6 records for DNS services

https://gerrit.wikimedia.org/r/930648

gerritbot added a comment.Jun 15 2023, 3:24 PM

Change 930654 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] acme_chief: openstack: codfw1dev: allow cloudservices2004-dev

https://gerrit.wikimedia.org/r/930654

gerritbot added a comment.Jun 15 2023, 3:27 PM

Change 930654 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] acme_chief: openstack: codfw1dev: allow cloudservices2004-dev

Reason:

will try something different

https://gerrit.wikimedia.org/r/930654

gerritbot added a comment.Jun 15 2023, 3:59 PM

Change 930661 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] acme_chief: openstack: codfw1dev: refresh LDAP certificates

https://gerrit.wikimedia.org/r/930661

gerritbot added a comment.Jun 15 2023, 4:06 PM

Change 930661 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] acme_chief: openstack: codfw1dev: refresh LDAP certificates

https://gerrit.wikimedia.org/r/930661

aborrero changed the status of subtask T338778: cloudservices2004-dev: reimage into new network setup from Open to In Progress.Jun 15 2023, 4:44 PM
aborrero updated the task description. (Show Details)Jun 16 2023, 11:41 AM
aborrero added a comment.Jun 16 2023, 11:49 AM

I just deleted netbox objects for IPv6:

  • 2620:0:860:2:208:80:153:47
  • 2620:0:860:2:208:80:153:50
gerritbot added a comment.Jun 16 2023, 12:01 PM

Change 930647 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: codfw1dev: drop IPv6 records for DNS services

https://gerrit.wikimedia.org/r/930647

gerritbot added a comment.Jun 16 2023, 12:10 PM

Change 930791 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: refresh ns0.openstack.codfw1dev address

https://gerrit.wikimedia.org/r/930791

gerritbot added a comment.Jun 16 2023, 12:10 PM

Change 930792 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimedia.cloud: fix delegation for codfw1dev

https://gerrit.wikimedia.org/r/930792

gerritbot added a comment.Jun 16 2023, 2:03 PM

Change 930791 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: refresh ns0.openstack.codfw1dev address

https://gerrit.wikimedia.org/r/930791

gerritbot added a comment.Jun 16 2023, 2:32 PM

Change 930792 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimedia.cloud: fix delegation for codfw1dev

https://gerrit.wikimedia.org/r/930792

gerritbot added a comment.Jun 16 2023, 2:34 PM

Change 930805 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] 57.15.185.in-addr.arpa: fix delegation

https://gerrit.wikimedia.org/r/930805

gerritbot added a comment.Jun 16 2023, 2:39 PM

Change 930805 merged by Arturo Borrero Gonzalez:

[operations/dns@master] 57.15.185.in-addr.arpa: fix delegation

https://gerrit.wikimedia.org/r/930805

gerritbot added a comment.Jun 16 2023, 2:45 PM

Change 930807 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: pdns: auth: service: drop IPv6 support

https://gerrit.wikimedia.org/r/930807

aborrero added a comment.Jun 16 2023, 2:54 PM

Deleting DNS name from the IPv6:

gerritbot added a comment.Jun 16 2023, 3:06 PM

Change 930808 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: drop unused ns-recursor[0,1] FQDNs

https://gerrit.wikimedia.org/r/930808

aborrero added a comment.Jun 16 2023, 3:09 PM

Deleting DNS entries for:

gerritbot added a comment.Jun 16 2023, 3:13 PM

Change 930808 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: drop unused ns-recursor[0,1] FQDNs

https://gerrit.wikimedia.org/r/930808

gerritbot added a comment.Jun 16 2023, 3:16 PM

Change 930648 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: eqiad1: drop IPv6 records for DNS services

https://gerrit.wikimedia.org/r/930648

gerritbot added a comment.Jun 16 2023, 3:18 PM

Change 930807 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: pdns: auth: service: drop IPv6 support

https://gerrit.wikimedia.org/r/930807

gerritbot added a comment.Jun 16 2023, 4:05 PM

Change 930817 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: make designate use cloud-private address by default

https://gerrit.wikimedia.org/r/930817

gerritbot added a comment.Jun 16 2023, 4:13 PM

Change 930817 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: make designate use cloud-private address by default

https://gerrit.wikimedia.org/r/930817

gerritbot added a comment.Jun 16 2023, 4:18 PM

Change 930818 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: designate: fix recursor_service_name

https://gerrit.wikimedia.org/r/930818

gerritbot added a comment.Jun 16 2023, 4:27 PM

Change 930820 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: codfw1dev: more hiera adjustements

https://gerrit.wikimedia.org/r/930820

gerritbot added a comment.Jun 16 2023, 4:28 PM

Change 930818 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: designate: fix recursor_service_name

https://gerrit.wikimedia.org/r/930818

gerritbot added a comment.Jun 16 2023, 4:29 PM

Change 930820 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: codfw1dev: more hiera adjustements

https://gerrit.wikimedia.org/r/930820

Stashbot added a comment.Jun 19 2023, 8:39 AM

Mentioned in SAL (#wikimedia-cloud) [2023-06-19T08:39:11Z] <arturo> update delegation for codfw1dev.wmcloud.org. to point to ns0/ns1 T307357

gerritbot added a comment.Jun 19 2023, 10:58 AM

Change 931239 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] dnsrecursor: introduce query-local-address parameter

https://gerrit.wikimedia.org/r/931239

gerritbot added a comment.Jun 19 2023, 12:30 PM

Change 931239 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] dnsrecursor: introduce query-local-address parameter

https://gerrit.wikimedia.org/r/931239

aborrero mentioned this in T339905: codfw1dev: LDAP setup needs a refresh so TLS works.Jun 20 2023, 10:28 AM
aborrero closed subtask T339905: codfw1dev: LDAP setup needs a refresh so TLS works as Resolved.Jun 20 2023, 11:37 AM
gerritbot added a comment.Jun 20 2023, 11:59 AM

Change 931589 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: refresh A for ns1.openstack.codfw1dev.wikimediacloud.org

https://gerrit.wikimedia.org/r/931589

gerritbot added a comment.Jun 20 2023, 12:02 PM

Change 931590 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: cleanup for ns.openstack.codfw1dev.wikimediacloud.org

https://gerrit.wikimedia.org/r/931590

aborrero closed subtask T338778: cloudservices2004-dev: reimage into new network setup as Resolved.Jun 20 2023, 12:12 PM
aborrero closed subtask T338938: codfw1dev: finish auth DNS server transition as Invalid.Jun 20 2023, 3:53 PM
gerritbot added a comment.Jun 20 2023, 4:17 PM

Change 930210 abandoned by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudservices2005-dev: fix DNS address

Reason:

merged with Ie9b38e58bca4fc152b54fa1efbb1e1d75040430c

https://gerrit.wikimedia.org/r/930210

gerritbot added a comment.Jun 21 2023, 8:59 AM

Change 931590 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: cleanup for ns.openstack.codfw1dev.wikimediacloud.org

https://gerrit.wikimedia.org/r/931590

gerritbot added a comment.Jun 21 2023, 9:09 AM

Change 931589 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: refresh A for ns1.openstack.codfw1dev.wikimediacloud.org

https://gerrit.wikimedia.org/r/931589

Maintenance_bot removed a project: Patch-For-Review.Jun 21 2023, 9:11 AM
aborrero changed the status of subtask T338779: cloudservices2005-dev: reimage into new network setup from Open to In Progress.Jun 21 2023, 2:48 PM
gerritbot added a comment.Jun 26 2023, 8:00 AM

Change 932794 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/software/netbox-extras@master] reports/network: ignore IPv6 for cloudservices boxes

https://gerrit.wikimedia.org/r/932794

gerritbot added a project: Patch-For-Review.Jun 26 2023, 8:01 AM
gerritbot added a comment.Jun 26 2023, 8:06 AM

Change 932794 merged by Arturo Borrero Gonzalez:

[operations/software/netbox-extras@master] reports/network: ignore IPv6 for cloudservices boxes

https://gerrit.wikimedia.org/r/932794

aborrero mentioned this in rOSNE69ccb67289ea: reports/network: ignore IPv6 for cloudservices boxes.Jun 26 2023, 8:07 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 26 2023, 8:10 AM
aborrero mentioned this in T340446: Cloud VPS Designate setup improvements.Jun 26 2023, 2:48 PM
aborrero added a parent task: T340446: Cloud VPS Designate setup improvements.
aborrero closed subtask T338779: cloudservices2005-dev: reimage into new network setup as Resolved.Jun 26 2023, 3:34 PM
aborrero moved this task from Doing to Radar on the User-aborrero board.
Andrew reopened subtask T338779: cloudservices2005-dev: reimage into new network setup as Open.Jun 26 2023, 7:46 PM
aborrero closed subtask T338779: cloudservices2005-dev: reimage into new network setup as Resolved.Jun 29 2023, 8:21 AM
aborrero added a subtask: T341966: prometheus in codfw can't reach cloudservices' pdns to fetch metrics.Jul 17 2023, 10:34 AM
fgiunchedi closed subtask T341966: prometheus in codfw can't reach cloudservices' pdns to fetch metrics as Resolved.Jul 18 2023, 9:57 AM
aborrero mentioned this in T342621: eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP.Jul 25 2023, 11:26 AM
aborrero added a subtask: T342621: eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP.Jul 25 2023, 11:35 AM
aborrero changed the status of subtask T342621: eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP from Open to Stalled.Aug 29 2023, 12:39 PM
aborrero changed the status of subtask T342621: eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP from Stalled to In Progress.Aug 30 2023, 9:57 AM
aborrero closed subtask T342621: eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP as Resolved.Sep 11 2023, 3:25 PM
gerritbot added a comment.Sep 14 2023, 2:48 PM

Change 957745 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: decom ns-recursor0.openstack.eqiad1.wikimediacloud.org

https://gerrit.wikimedia.org/r/957745

gerritbot added a project: Patch-For-Review.Sep 14 2023, 2:48 PM
aborrero closed this task as Resolved.Sep 20 2023, 10:25 AM
aborrero claimed this task.
gerritbot added a comment.Sep 26 2023, 9:27 AM

Change 961054 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] cr-cloud: Remove unused terms

https://gerrit.wikimedia.org/r/961054

gerritbot added a comment.Sep 26 2023, 12:07 PM

Change 961054 merged by jenkins-bot:

[operations/homer/public@master] cr-cloud: Remove unused terms

https://gerrit.wikimedia.org/r/961054

cmooney added a subtask: T347858: Manage WMCS codfw public service VIP DNS from Netbox.Oct 2 2023, 12:39 PM
cmooney closed subtask T347858: Manage WMCS codfw public service VIP DNS from Netbox as Resolved.Oct 2 2023, 12:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL