Page MenuHomePhabricator
Create Task
Maniphest T247972

Cloud DNS: fix inconsistent ownership of reverse domains for openstack floating ip networks
Closed, ResolvedPublic
Actions

Description

We have 1 floating IP network (CIDR) in each openstack deployment:

  • 185.15.56.0/25 -- eqiad1. Zone 56.15.185.in-addr.arpa. owned by designate @ eqiad1 (wmflabsdotorg project)
  • 185.15.57.0/29 -- codfw1dev. Zone 57.15.185.in-addr.arpa owned by prod DNS servers.

In markmonitor:

arturo@endurance:~ $ whois 57.15.185.in-addr.arpa | grep "nserver"
nserver:        ns0.wikimedia.org
nserver:        ns1.wikimedia.org
nserver:        ns2.wikimedia.org
arturo@endurance:~ $ whois 56.15.185.in-addr.arpa | grep "nserver"
nserver:        cloud-ns0.wikimedia.org
nserver:        cloud-ns1.wikimedia.org

This is inconsistent. Probably the right thing to do is to delegate the /29 in codfw to designate @ codw1dev and update markmonitor.
The nserver entry requires also an update to ns[0-1].openstack.<deployment>.wikimediacloud.org, similar to what is happening in T247971: Cloud DNS: update markmonitor entries.

Project ownership should probably be set to cloudinfra (instead of wmflabsdotorg) to be consistent with https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS
This may or may not require changes in other scripts or mechanism (proxies generate PTR records?)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
57.15.185-in-addr.arpa: refresh zone name of the delegationoperations/dnsmaster+10 -10
openstack: codfw1dev: designate: update zone name for floating IP subnetoperations/puppetproduction+1 -1
templates: add reverse zone for 185.12.57.0/24 including cloud delegationoperations/dnsmaster+21 -0
57.15.185.in-addr.arpa: drop zoneoperations/dnsmaster+0 -10
Customize query in gerrit

Related Objects
Search...

Event Timeline

• aborrero created this task.Mar 18 2020, 1:33 PM
• aborrero triaged this task as Medium priority.Mar 18 2020, 1:35 PM
• aborrero updated the task description. (Show Details)
• aborrero moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.

@Andrew ping me if you prefer me to handle this.

Andrew added a parent task: T247971: Cloud DNS: update markmonitor entries.Mar 25 2020, 6:35 PM
Krenair subscribed.EditedMar 25 2020, 10:48 PM

gotcha to be aware of:
the /25 you list for eqiad1 is accurate in that it looks like that's what neutron will allocate from (based on modules/openstack/templates/bootstrap/neutron/neutron_seed.sh.erb), but actually the whole /24 is reserved for the purpose. so this existing thing is okay.
The 57.15.185.in-addr.arpa zone is for the whole of 185.15.57.0/24 - note the /24, not /29. I'm not sure what's been reserved, this seems to be missing from modules/network/data/data.yaml? (and possibly other places?)

I'm not sure if we have anything updating that zone other than wmcs-dns-floating-ip-updater.py by the way

Andrew mentioned this in T247971: Cloud DNS: update markmonitor entries.Mar 31 2020, 8:08 PM
• aborrero added a subscriber: ayounsi.Apr 1 2020, 11:30 AM

The whole /24 is marked in Netbox as cloud-codfw: https://netbox.wikimedia.org/ipam/prefixes/5/

Please @ayounsi confirm this range is allocated entirely for us.

ayounsi added a comment.Apr 2 2020, 5:01 PM

It is reserved for you. And allocated as your needs grow. So far 185.15.57.0/29. As we're trying to be extra careful with v4 IPs.

I updated Netbox so it matches it's eqiad's equivalent: https://netbox.wikimedia.org/ipam/prefixes/1/prefixes/

• aborrero added a comment.Apr 6 2020, 5:29 PM

OK thanks for confirming! I will:

  • drop the 57.15.185.in-addr.arpa zone zone from production DNS
  • add that reverse zone to designate / cloudinfra-codfw1dev
gerritbot added a comment.Apr 6 2020, 5:32 PM

Change 586409 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/dns@master] 57.15.185.in-addr.arpa: drop zone

https://gerrit.wikimedia.org/r/586409

gerritbot added a project: Patch-For-Review.Apr 6 2020, 5:32 PM
Stashbot added a comment.Apr 6 2020, 5:39 PM

Mentioned in SAL (#wikimedia-cloud) [2020-04-06T17:39:07Z] <arturo> [codfw1dev] openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description "floating IPs subnet" 57.15.185.in-addr.arpa. (T247972)

Stashbot added a comment.Apr 6 2020, 5:42 PM

Mentioned in SAL (#wikimedia-cloud) [2020-04-06T17:42:18Z] <arturo> [codfw1dev] transferred DNS zone 57.15.185.in-addr.arpa. to the cloudinfra-codfw1dev project (T247972)

gerritbot added a comment.Apr 6 2020, 5:43 PM

Change 586409 merged by Arturo Borrero Gonzalez:
[operations/dns@master] 57.15.185.in-addr.arpa: drop zone

https://gerrit.wikimedia.org/r/586409

• aborrero closed this task as Resolved.Apr 6 2020, 5:44 PM

This seems OK now! thanks everyone!

Maintenance_bot removed a project: Patch-For-Review.Apr 6 2020, 6:10 PM
Andrew reopened this task as Open.EditedApr 6 2020, 8:00 PM

I still see:

andrew@buster:~$ whois 56.15.185.in-addr.arpa | grep nserver
nserver: cloud-ns0.wikimedia.org
nserver: cloud-ns1.wikimedia.org
andrew@buster:~$ whois 57.15.185.in-addr.arpa | grep nserver
nserver: ns0.wikimedia.org
nserver: ns1.wikimedia.org
nserver: ns2.wikimedia.org

I briefly thought that we needed to update markmonitor for those, but @Krenair pointed out that it's a RIPE thing. @ayounsi, is that something you can update with them? It should be

57.15.185.in-addr.arpa:

ns0.openstack.codfw1dev.wikimediacloud.org 208.80.153.76

56.15.185.in-addr.arpa:

ns0.openstack.eqiad1.wikimediacloud.org 208.80.154.135

ns1.openstack.eqiad1.wikimediacloud.org 208.80.154.11

Andrew added a comment.Apr 6 2020, 8:14 PM

I chatted with Faidon about this a bit and he's going to check in with Arzhel about 57.15.185.in-addr.arpa

ayounsi added a comment.May 12 2020, 4:33 PM

I started to look into that:

Trying to create the following domain object:

domain:         0-7.57.15.185.in-addr.arpa
descr:          Wikimedia_cloud_codfw
admin-c:        FAID1-RIPE
admin-c:        MBE96-RIPE
tech-c:         FAID1-RIPE
tech-c:         MBE96-RIPE
tech-c:         AY3199-RIPE
zone-c:         WMF-RIPE
nserver:        ns0.openstack.codfw1dev.wikimediacloud.org
mnt-by:         WIKIMEDIA-MNT
source:         RIPE

Returns a create error:

No response from nameserver for zone 57.15.185.in-addr.arpa when trying to fetch glue.
Not enough data about 0-7.57.15.185.in-addr.arpa was found to be able to run tests.

Can you double check that the NS is properly configured?

About 56.15.185.in-addr.arpa is it ok to do it anytime or do I need to sync up with you?

Andrew added a comment.May 18 2020, 4:46 PM

I fixed at least one thing with the 57.15.185.in-addr.arpa zone (the SOA was pointing to a currently broken resolver).

The 56. domain can be updated whenever, but I'd appreciate a ping when you do it.

Thanks!

Stashbot added a comment.May 18 2020, 5:14 PM

Mentioned in SAL (#wikimedia-operations) [2020-05-18T17:14:17Z] <XioNoX> update domain object for 56.15.185.in-addr.arpa - T247972

ayounsi added a comment.May 18 2020, 5:16 PM
eqiad
domain:         56.15.185.in-addr.arpa
descr:          Wikimedia_cloud_eqiad
admin-c:        FAID1-RIPE
admin-c:        MBE96-RIPE
tech-c:         FAID1-RIPE
tech-c:         MBE96-RIPE
tech-c:         AY3199-RIPE
zone-c:         WMF-RIPE
nserver:        ns0.openstack.eqiad1.wikimediacloud.org
nserver:        ns1.openstack.eqiad1.wikimediacloud.org
mnt-by:         WIKIMEDIA-MNT
source:         RIPE

The following object(s) were processed SUCCESSFULLY:
Modify SUCCEEDED: [domain] 56.15.185.in-addr.arpa

codfw still fails with the same error.

ayounsi added a subscriber: jbond.May 19 2020, 7:24 AM

I spent some time digging through the RIPE doc, but can't find any clear answer for T247972#6130041.
@jbond do you have any idea?

Otherwise I'll reach out to the RIPE support.

ayounsi added a comment.May 19 2020, 4:28 PM

Discussed it with John, so 57.15.185.in-addr.arpa is configured to have ns1/2/3.wikimedia.org as NS. Which is correct.

The next steps is to delegate 0-7.57.15.185.in-addr.arpa to ns0.openstack.codfw1dev.wikimediacloud.org
Examples are in the RFC https://tools.ietf.org/html/rfc2317
Unfortunately I don't know enough about DNS to help with the exact config needed though.

Andrew reassigned this task from Andrew to • aborrero.May 19 2020, 5:56 PM

Reassigning to arturo in case he knows how to proceed and/or thinks that we don't actually need to do this :)

• aborrero added a comment.May 20 2020, 9:00 AM

ok, will take a look soon.

gerritbot added a comment.May 20 2020, 11:26 AM

Change 597514 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/dns@master] templates: add reverse zone for 185.12.57.0/24 including cloud delegation

https://gerrit.wikimedia.org/r/597514

gerritbot added a project: Patch-For-Review.May 20 2020, 11:26 AM
gerritbot added a comment.May 20 2020, 3:04 PM

Change 597514 merged by Arturo Borrero Gonzalez:
[operations/dns@master] templates: add reverse zone for 185.12.57.0/24 including cloud delegation

https://gerrit.wikimedia.org/r/597514

Maintenance_bot removed a project: Patch-For-Review.May 20 2020, 3:11 PM
• aborrero added a comment.May 20 2020, 3:59 PM

TODO: verify that we can actually resolve stuff like dig 1.57.15.185.in-addr.arpa. Will probably need to take a look at designate @ codfw1dev

jbond added a comment.May 21 2020, 9:04 AM

I noticed that the ns0.openstack.codfw1dev.wikimediacloud.org. servers are configured as name servers for the 57.15.185.in-addr.arpa. zone. however you actually need to configure the 0/29.57.15.185.in-addr.arpa. zone

• aborrero added a comment.May 25 2020, 12:47 PM

mmmm

root@cloudcontrol2001-dev:~# openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description "floating IPs subnet" 0/29.57.15.185.in-addr.arpa.
Provided object is not valid. Got a ValueError error with message Domain 0/29.57.15.185.in-addr.arpa. is not match

investigating why designate complains about this.

• aborrero added a comment.May 25 2020, 12:55 PM

Apparently the string gets evaluated against this regex: RE_ZONENAME = r'^(?!.{255,})(?:(?!\-)[A-Za-z0-9_\-]{1,63}(?<!\-)\.)+\Z' which doesn't allow the 0/29.

Stashbot added a comment.May 25 2020, 4:35 PM

Mentioned in SAL (#wikimedia-cloud) [2020-05-25T16:35:57Z] <arturo> [codfw1dev] created zone 0-29.57.15.185.in-addr.arpa. (T247972)

gerritbot added a comment.May 25 2020, 4:57 PM

Change 598502 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: codfw1dev: designate: update zone name for floating IP subnet

https://gerrit.wikimedia.org/r/598502

gerritbot added a project: Patch-For-Review.May 25 2020, 4:57 PM
gerritbot added a comment.May 25 2020, 4:59 PM

Change 598503 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/dns@master] 57.15.185-in-addr.arpa: refresh zone name of the delegation

https://gerrit.wikimedia.org/r/598503

gerritbot added a comment.May 25 2020, 4:59 PM

Change 598502 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: codfw1dev: designate: update zone name for floating IP subnet

https://gerrit.wikimedia.org/r/598502

gerritbot added a comment.May 25 2020, 5:00 PM

Change 598503 merged by Arturo Borrero Gonzalez:
[operations/dns@master] 57.15.185-in-addr.arpa: refresh zone name of the delegation

https://gerrit.wikimedia.org/r/598503

Maintenance_bot removed a project: Patch-For-Review.May 25 2020, 5:10 PM
• aborrero closed this task as Resolved.May 25 2020, 5:19 PM

Ok, after a bit of back and forth, this works now:

arturo@endurance:~$ dig -x 185.15.57.1 +short
1.0-29.57.15.185.in-addr.arpa.
test.codfw1dev.wmcloud.org.
arturo@endurance:~$ dig -x 185.15.57.2 +short
2.0-29.57.15.185.in-addr.arpa.
test.codfw1dev.wmcloud.org.

When I saw openstack not accepting the / character I rushed to fill a bug report: https://bugs.launchpad.net/designate/+bug/1880583
I later disovered that the same can be done with the - character, an that's what I did.

BTW, did a couple of updates to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate because Designate seems a bit more robust in openstack rocky :-)

• aborrero removed a parent task: T245173: CloudVPS: DNS improvements.Jun 25 2020, 11:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits