Undocumented IP on WMCS network
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ayounsi
	Aug 23 2022, 5:42 AM

Description

I received a report from Qrator saying that portmap was exposed on 185.15.57.20.

I wasn't able to reproduce it (and it says "Last seen: 2022-08-22 14:00:34") but that IP is not documented in DNS nor Netbox.

That IP replies to pings and is routed to wan.cloudgw.codfw1dev.wikimediacloud.org.

185.15.57.16/29    *[Static/5] 3w5d 09:35:28
                    > to 208.80.153.190 via ae2.2120

Please make sure this IP and any others are documented in Netbox, DNS or ideally both. And double check that portmap isn't exposed as it's a DDoS vector.

Details

	Subject	Repo	Branch	Lines +/-
	Sub-delegation of reverse DNS entries for 185.15.57.16/29 to WMCS	operations/dns	master	+12 -0
	wikimediacloud.org: do not use CNAMEs for nsX addresses	operations/dns	master	+9 -4

Customize query in gerrit

Related Objects

Mentioned Here: T313977: More public IPs for codfw1dev

Event Timeline

ayounsi triaged this task as High priority.Aug 23 2022, 5:42 AM

ayounsi created this task.

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptAug 23 2022, 5:42 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Maintenance_bot added a project: SRE.Aug 23 2022, 5:45 AM

ayounsi updated the task description. (Show Details)Aug 23 2022, 5:46 AM

RhinosF1 subscribed.Aug 23 2022, 7:40 AM

Peachey88 subscribed.Aug 23 2022, 7:59 AM

I think that this might be the experiments that we have been doing with Magnum, ping @Andrew, @rook

ayounsi moved this task from Backlog to Watching on the netops board.Aug 23 2022, 10:20 AM

Yeah, this is associated with the testing we're doing with magnum. It's part of 185.15.57.16/29 which was assigned to codfw1dev in T313977
How does one document them?

Andrew claimed this task.Aug 24 2022, 4:34 PM

This additional range was set up by @cmooney -- Cathal, is this something you can document as needed?

Thanks @Andrew. It seems I failed to update the description for the /29 in Netbox, so let me correct that. Can you suggest an appropriate description for the 185.15.57.16/29 network overall?

In terms of individual IPs it's not really possible for us to know which you have set up and where. So documenting the IP as @ayounsi requested is something that can only be done by the cloud team.

I'm not sure what agreed process might exist here (or not). I see for the other cloud ranges there are no individual IPs documented:

185.15.56.0/25
185.15.57.0/29

In general SRE teams would manage this themselves, but this is slightly different as entire subnets are delegated rather than single IPs. Many are used for NAT pools I know, which again is somewhat unique. If there is no existing agreement on how to handle these we should maybe discuss at our next meeting.

In terms of the reverse DNS I can see in Eqiad the entire /24 is delegated to the cloud DNS servers:

cathal@officepc:~$ dig +noall +authority NS 56.15.185.in-addr.arpa. @pri.authdns.ripe.net.
56.15.185.in-addr.arpa. 86400   IN      NS      ns0.openstack.eqiad1.wikimediacloud.org.
56.15.185.in-addr.arpa. 86400   IN      NS      ns1.openstack.eqiad1.wikimediacloud.org.

So for that perhaps we need to investigate our options for delegating sub-ranges in codfw? I'd need to check our options there in terms of what our authdns can support, but if possible it would make it equivalent to eqiad.

The label should just be 'public floating IPs for cloud-vps codfw1dev' -- by their very nature the actual use of any particular IP will shift over time based on self-serve use.

As for eqiad, I see these ranges allocated for floating IPs:

185.15.56.0/25
185.15.56.240/29
185.15.56.236/30

That is not quite the whole /24 but it's realistic to describe that range as 'public floating IPs for cloud-vps eqiad1'

As for DNS... there are equivalent servers in Dallas that should be responsible for the range in codfw1dev: ns0.openstack.codfw1dev.wikimediacloud.org and ns1.openstack.codfw1dev.wikimediacloud.org.

Thanks Andrew, I've updated the description for the codfw range now.

In terms of DNS I don't seem to get any PTR records back for the ranges in codfw:

cathal@officepc:~$ for i in {0..255}; do dig +noall +answer -x 185.15.57.$i @ns0.openstack.codfw1dev.wikimediacloud.org; done
cathal@officepc:~$

Which is quite different to the case for eqiad:

cathal@officepc:~$ for i in {0..255}; do dig +noall +answer -x 185.15.56.$i @ns0.openstack.eqiad1.wikimediacloud.org.; done;
1.56.15.185.in-addr.arpa. 3600  IN      PTR     nat.cloudgw.eqiad1.wikimediacloud.org.
2.56.15.185.in-addr.arpa. 3600  IN      PTR     aliastest.testlabs.wmflabs.org.
2.56.15.185.in-addr.arpa. 3600  IN      PTR     pooltest.testlabs.wmflabs.org.
2.56.15.185.in-addr.arpa. 3600  IN      PTR     dnstest.testlabs.wmflabs.org.
2.56.15.185.in-addr.arpa. 3600  IN      PTR     gtirloni-stretch-01.testlabs.wmflabs.org.
2.56.15.185.in-addr.arpa. 3600  IN      PTR     abogott-test.testlabs.wmflabs.org.
3.56.15.185.in-addr.arpa. 3600  IN      PTR     ntp-01.wmflabs.org.
3.56.15.185.in-addr.arpa. 3600  IN      PTR     ntp-03.cloudinfra.wmflabs.org.
3.56.15.185.in-addr.arpa. 3600  IN      PTR     instance-ntp-03.cloudinfra.wmflabs.org.
3.56.15.185.in-addr.arpa. 3600  IN      PTR     ntp-01.cloudinfra.wmflabs.org.
5.56.15.185.in-addr.arpa. 3600  IN      PTR     instance-wikiloop3.wikiloop.wmflabs.org.
7.56.15.185.in-addr.arpa. 3600  IN      PTR     instance-mars.wikisp.wmflabs.org.
8.56.15.185.in-addr.arpa. 3600  IN      PTR     harbor.toolsbeta.wmflabs.org.
8.56.15.185.in-addr.arpa. 3600  IN      PTR     instance-toolsbeta-harborweb-2.toolsbeta.wmflabs.org.
9.56.15.185.in-addr.arpa. 3600  IN      PTR     instance-deployment-parsoid12.deployment-prep.wmflabs.org.
9.56.15.185.in-addr.arpa. 3600  IN      PTR     parsoid-external-ci-access.beta.wmflabs.org.
11.56.15.185.in-addr.arpa. 3600 IN      PTR     toolforge.org.
11.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-tools-proxy-06.tools.wmflabs.org.
12.56.15.185.in-addr.arpa. 3600 IN      PTR     mail.toolsbeta.wmflabs.org.
12.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-toolsbeta-mail-01.toolsbeta.wmflabs.org.
13.56.15.185.in-addr.arpa. 3600 IN      PTR     eqiad1.bastion.wmflabs.org.
14.56.15.185.in-addr.arpa. 3600 IN      PTR     bastion-restricted.wmflabs.org.
14.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-bastion-restricted-eqiad1-02.bastion.wmflabs.org.
14.56.15.185.in-addr.arpa. 3600 IN      PTR     restricted.bastion.wmcloud.org.
14.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-bastion-restricted-eqiad1-01.bastion.wmcloud.org.
14.56.15.185.in-addr.arpa. 3600 IN      PTR     eqiad1-restricted.bastion.wmflabs.org.
14.56.15.185.in-addr.arpa. 3600 IN      PTR     eqiad1-restricted.bastion.wmcloud.org.
14.56.15.185.in-addr.arpa. 3600 IN      PTR     restricted.bastion.wmflabs.org.
16.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-bastion-eqiad1-02.bastion.wmcloud.org.
17.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-enc-1.cloudinfra.wmflabs.org.
17.56.15.185.in-addr.arpa. 3600 IN      PTR     puppet-enc.cloudinfra.wmcloud.org.
18.56.15.185.in-addr.arpa. 3600 IN      PTR     mx-out03.wmcloud.org.
18.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-mx-out03.cloudinfra.wmflabs.org.
19.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-mx-out04.cloudinfra.wmflabs.org.
19.56.15.185.in-addr.arpa. 3600 IN      PTR     mx-out04.wmcloud.org.
20.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-cvn-app9.cvn.wmflabs.org.
21.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-cvn-app8.cvn.wmflabs.org.
22.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-cyberbot-exec-iabot-01.cyberbot.wmflabs.org.
24.56.15.185.in-addr.arpa. 3600 IN      PTR     rc.huggle.wmcloud.org.
24.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-xmlrcs.huggle.wmflabs.org.
26.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-mwstake.mwstake.wmflabs.org.
27.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-ntp-04.cloudinfra.wmflabs.org.
27.56.15.185.in-addr.arpa. 3600 IN      PTR     ntp-04.cloudinfra.wmflabs.org.
27.56.15.185.in-addr.arpa. 3600 IN      PTR     ntp-02.wmflabs.org.
27.56.15.185.in-addr.arpa. 3600 IN      PTR     ntp-02.cloudinfra.wmflabs.org.
28.56.15.185.in-addr.arpa. 3600 IN      PTR     lists.wmcloud.org.
29.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-cyberbot-exec-iabot-02.cyberbot.wmflabs.org.
30.56.15.185.in-addr.arpa. 3600 IN      PTR     matrix.wmflabs.org.
30.56.15.185.in-addr.arpa. 3600 IN      PTR     synapse.matrix.wmflabs.org.
31.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-webservices.getstarted.wmflabs.org.
32.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-pub2.wikiapiary.wmflabs.org.
33.56.15.185.in-addr.arpa. 3600 IN      PTR     mail.beta.wmflabs.org.
34.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-deployment-ircd02.deployment-prep.wmflabs.org.
34.56.15.185.in-addr.arpa. 3600 IN      PTR     irc.beta.wmflabs.org.
35.56.15.185.in-addr.arpa. 3600 IN      PTR     upload.beta.wmflabs.org.
35.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-deployment-cache-upload06.deployment-prep.wmflabs.org.
35.56.15.185.in-addr.arpa. 3600 IN      PTR     upload.wikimedia.beta.wmflabs.org.
36.56.15.185.in-addr.arpa. 3600 IN      PTR     m.wikidata.beta.wmflabs.org.
36.56.15.185.in-addr.arpa. 3600 IN      PTR     config-master.wikimedia.beta.wmflabs.org.
36.56.15.185.in-addr.arpa. 3600 IN      PTR     wikifunctions.beta.wmflabs.org.
36.56.15.185.in-addr.arpa. 3600 IN      PTR     m.wikifunctions.beta.wmflabs.org.
36.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-deployment-cache-text06.deployment-prep.wmflabs.org.
36.56.15.185.in-addr.arpa. 3600 IN      PTR     wikidata.beta.wmflabs.org.
36.56.15.185.in-addr.arpa. 3600 IN      PTR     beta.wmflabs.org.
39.56.15.185.in-addr.arpa. 3600 IN      PTR     oxygen.rcm.wmflabs.org.
39.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-oxygen.rcm.wmflabs.org.
39.56.15.185.in-addr.arpa. 3600 IN      PTR     irc.rcm.wmflabs.org.
40.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-bastion-eqiad1-04.bastion.wmflabs.org.
40.56.15.185.in-addr.arpa. 3600 IN      PTR     secondary.bastion.wmcloud.org.
40.56.15.185.in-addr.arpa. 3600 IN      PTR     secondary.bastion.wmflabs.org.
41.56.15.185.in-addr.arpa. 3600 IN      PTR     beta.math.wmflabs.org.
41.56.15.185.in-addr.arpa. 3600 IN      PTR     mardi.math.wmflabs.org.
41.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-math19.math.wmflabs.org.
43.56.15.185.in-addr.arpa. 3600 IN      PTR     lists.wmcloud.org.
43.56.15.185.in-addr.arpa. 3600 IN      PTR     polymorphic.lists.wmcloud.org.
43.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-mailman03.mailman.wmflabs.org.
44.56.15.185.in-addr.arpa. 3600 IN      PTR     vmbuilder-trusty.openstack.wmflabs.org.
45.56.15.185.in-addr.arpa. 3600 IN      PTR     bootstrap-vz-jessie.openstack.wmflabs.org.
45.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-cloud-bootstrapvz-stretch.openstack.wmflabs.org.
46.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-medbox4-iiab.iiab.wmflabs.org.
48.56.15.185.in-addr.arpa. 3600 IN      PTR     login-stretch.tools.wmflabs.org.
49.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-proxy-03.project-proxy.wmflabs.org.
50.56.15.185.in-addr.arpa. 3600 IN      PTR     stretch-dev.tools.wmflabs.org.
53.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-clouddb-wikireplicas-proxy-2.clouddb-services.wmflabs.org.
53.56.15.185.in-addr.arpa. 3600 IN      PTR     staticproxy-1.testlabs.wmflabs.org.
53.56.15.185.in-addr.arpa. 3600 IN      PTR     staticproxy-3.testlabs.wmflabs.org.
53.56.15.185.in-addr.arpa. 3600 IN      PTR     staticproxy-2.testlabs.wmflabs.org.
54.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-google-api-proxy-03.google-api-proxy.wmflabs.org.
55.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-maps-proxy-03.project-proxy.wmflabs.org.
57.56.15.185.in-addr.arpa. 3600 IN      PTR     paws.wmflabs.org.
57.56.15.185.in-addr.arpa. 3600 IN      PTR     paws.wmcloud.org.
57.56.15.185.in-addr.arpa. 3600 IN      PTR     paws-public.wmflabs.org.
58.56.15.185.in-addr.arpa. 3600 IN      PTR     paws-beta.wmflabs.org.
59.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-sec-utils-bullseye.security-tools.wmflabs.org.
60.56.15.185.in-addr.arpa. 3600 IN      PTR     tools.wmflabs.org.
60.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-tools-legacy-redirector.tools.wmflabs.org.
61.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-tools-checker-04.tools.wmflabs.org.
61.56.15.185.in-addr.arpa. 3600 IN      PTR     checker.tools.wmflabs.org.
62.56.15.185.in-addr.arpa. 3600 IN      PTR     relay.toolserver.org.
62.56.15.185.in-addr.arpa. 3600 IN      PTR     toolserver.org.
62.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-toolserver-proxy-01.tools.wmflabs.org.
63.56.15.185.in-addr.arpa. 3600 IN      PTR     mail.tools.wmflabs.org.
63.56.15.185.in-addr.arpa. 3600 IN      PTR     mailsender.tools.wmflabs.org.
63.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-tools-mail-03.tools.wmflabs.org.
63.56.15.185.in-addr.arpa. 3600 IN      PTR     mail.tools.wmcloud.org.
64.56.15.185.in-addr.arpa. 3600 IN      PTR     puppetmaster.cloudinfra.wmflabs.org.
64.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-cloud-puppetmaster-03.cloudinfra.wmflabs.org.
66.56.15.185.in-addr.arpa. 3600 IN      PTR     login-buster.toolforge.org.
66.56.15.185.in-addr.arpa. 3600 IN      PTR     login.tools.wmflabs.org.
66.56.15.185.in-addr.arpa. 3600 IN      PTR     bastion.toolforge.org.
66.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-tools-sgebastion-10.tools.wmflabs.org.
67.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-tools-docker-registry-05.tools.wmflabs.org.
67.56.15.185.in-addr.arpa. 3600 IN      PTR     docker-registry.tools.wmflabs.org.
69.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-hupu.wikidocumentaries.wmflabs.org.
70.56.15.185.in-addr.arpa. 3600 IN      PTR     gerrit.devtools.wmflabs.org.
70.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-gerrit-prod-1001.devtools.wmflabs.org.
72.56.15.185.in-addr.arpa. 3600 IN      PTR     stream.meet.wmcloud.org.
72.56.15.185.in-addr.arpa. 3600 IN      PTR     meet.wmcloud.org.
72.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-jitsi04.meet.wmflabs.org.
74.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-deployment-docker-wikifunctions01.deployment-prep.wmflabs.org.
75.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-pontoon-frontend-02.monitoring.wmflabs.org.
79.56.15.185.in-addr.arpa. 3600 IN      PTR     gitlab.devtools.wmflabs.org.
79.56.15.185.in-addr.arpa. 3600 IN      PTR     gitlab.devtools.wmcloud.org.
81.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-wm-bot.wm-bot.wmflabs.org.
81.56.15.185.in-addr.arpa. 3600 IN      PTR     wm-bot.wm-bot.wmcloud.org.
84.56.15.185.in-addr.arpa. 3600 IN      PTR     docker-registry.toolsbeta.wmflabs.org.
85.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-ircwebchat.ircwebchat.wmflabs.org.
87.56.15.185.in-addr.arpa. 3600 IN      PTR     primary.bastion.wmflabs.org.
87.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-bastion-eqiad1-01.bastion.wmcloud.org.
87.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-bastion-eqiad1-03.bastion.wmflabs.org.
87.56.15.185.in-addr.arpa. 3600 IN      PTR     bastion.wmflabs.org.
87.56.15.185.in-addr.arpa. 3600 IN      PTR     bastion.wmcloud.org.
87.56.15.185.in-addr.arpa. 3600 IN      PTR     primary.bastion.wmcloud.org.
87.56.15.185.in-addr.arpa. 3600 IN      PTR     eqiad1.bastion.wmcloud.org.
93.56.15.185.in-addr.arpa. 3600 IN      PTR     instance-node-1.canasta.wmflabs.org.
102.56.15.185.in-addr.arpa. 3600 IN     PTR     instance-mwcurator.mwoffliner.wmflabs.org.
121.56.15.185.in-addr.arpa. 3600 IN     PTR     instance-clouddb-wikireplicas-proxy-1.clouddb-services.wmflabs.org.
122.56.15.185.in-addr.arpa. 3600 IN     PTR     dev-buster.toolforge.org.
122.56.15.185.in-addr.arpa. 3600 IN     PTR     instance-tools-sgebastion-11.tools.wmflabs.org.
122.56.15.185.in-addr.arpa. 3600 IN     PTR     dev.tools.wmflabs.org.
122.56.15.185.in-addr.arpa. 3600 IN     PTR     dev.toolforge.org.
126.56.15.185.in-addr.arpa. 3600 IN     PTR     instance-apollo.wikisp.wmflabs.org.
237.56.15.185.in-addr.arpa. 3600 IN     PTR     virt.cloudgw.eqiad1.wikimediacloud.org.
238.56.15.185.in-addr.arpa. 3600 IN     PTR     cloudinstances2b-gw.openstack.eqiad1.wikimediacloud.org.
241.56.15.185.in-addr.arpa. 3600 IN     PTR     vrrp-gw-1120.eqiad1.wikimediacloud.org.
242.56.15.185.in-addr.arpa. 3600 IN     PTR     irb-1120.cloudsw1-c8-eqiad.eqiad1.wikimediacloud.org.
243.56.15.185.in-addr.arpa. 3600 IN     PTR     irb-1120.cloudsw1-d5-eqiad.eqiad1.wikimediacloud.org.
244.56.15.185.in-addr.arpa. 3600 IN     PTR     wan.cloudgw.eqiad1.wikimediacloud.org.
245.56.15.185.in-addr.arpa. 3600 IN     PTR     cloudgw1001.eqiad1.wikimediacloud.org.
246.56.15.185.in-addr.arpa. 3600 IN     PTR     cloudgw1002.eqiad1.wikimediacloud.org.
cathal@officepc:~$

I'm not sure if anything more needs to be done in codfw to make it return records for the ranges there? But either way it seems there are cloud DNS servers in codfw, and I gather you're happy for us to delegate the reverse ranges if possible? If so I'll investigate if that's possible for us to do. Cheers.

Change 826803 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Sub-delegation of reverse DNS entries for 185.15.57.16/29 to WMCS

https://gerrit.wikimedia.org/r/826803

gerritbot added a project: Patch-For-Review.Aug 26 2022, 9:30 AM

Added above patch to delegate this range to the WMCS name servers. I hadn't checked the naming convention previously, I do actually see entries for the 185.15.57.0/29 range on those servers:

cathal@officepc:~$ dig +short SOA 0-29.57.15.185.in-addr.arpa. @ns0.openstack.codfw1dev.wikimediacloud.org.
ns1.openstack.codfw1dev.wikimediacloud.org. root.wmflabs.org. 1650969226 3547 600 86400 3600

cathal@officepc:~$ for i in {0..7}; do dig +noall +answer PTR $i.0-29.57.15.185.in-addr.arpa. @ns0.openstack.codfw1dev.wikimediacloud.org.; done
1.0-29.57.15.185.in-addr.arpa. 3600 IN  PTR     nat.cloudgw.codfw1dev.wikimediacloud.org.
2.0-29.57.15.185.in-addr.arpa. 3600 IN  PTR     bastion.bastioninfra-codfw1dev.codfw1dev.wmcloud.org.
5.0-29.57.15.185.in-addr.arpa. 3600 IN  PTR     manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org.

But I think the new zone, 16-29.57.15.185.in-addr.arpa. needs to be set up there:

cathal@officepc:~$ for i in {16..23}; do dig +noall +answer PTR $i.16-29.57.15.185.in-addr.arpa. @ns0.openstack.codfw1dev.wikimediacloud.org.; done
cathal@officepc:~$
cathal@officepc:~$ dig SOA 16-29.57.15.185.in-addr.arpa. @ns0.openstack.codfw1dev.wikimediacloud.org.

; <<>> DiG 9.18.1-1ubuntu1.1-Ubuntu <<>> SOA 16-29.57.15.185.in-addr.arpa. @ns0.openstack.codfw1dev.wikimediacloud.org.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 57416
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;16-29.57.15.185.in-addr.arpa.  IN      SOA

;; AUTHORITY SECTION:
57.15.185.in-addr.arpa. 3600    IN      SOA     ns1.openstack.codfw1dev.wikimediacloud.org. root.wmflabs.org. 1650969224 3520 600 86400 3600

;; Query time: 132 msec
;; SERVER: 2620:0:860:2:208:80:153:43#53(ns0.openstack.codfw1dev.wikimediacloud.org.) (UDP)
;; WHEN: Fri Aug 26 11:02:46 IST 2022
;; MSG SIZE  rcvd: 148

Also just a note on the setup of the WMCS DNS in general.

It seems BIND won't resolve any of these names because the CNAMEs on the nsX.openstack hostnames. I get a SERVFAIL locally because of this, and spotted the following in my local log:

Aug 26 12:06:29 nbgw named[53518]: 26-Aug-2022 12:06:29.586 cname: info: skipping nameserver 'ns0.openstack.codfw1dev.wikimediacloud.org' because it is a CNAME, while resolving '0-29.57.15.185.in-addr.arpa/NS'

The cause is a relatively archaic DNS rule that hostnames references in NS entries should point directly to A/AAAA records, and not CNAMEs. Many resolvers ignore this, and it works, but Bind is not uncommon and the fact it cannot resolve these names may be a concern.

In T315955#8188444, @cmooney wrote:
Also just a note on the setup of the WMCS DNS in general.

It seems BIND won't resolve any of these names because the CNAMEs on the nsX.openstack hostnames. I get a SERVFAIL locally because of this, and spotted the following in my local log:
Aug 26 12:06:29 nbgw named[53518]: 26-Aug-2022 12:06:29.586 cname: info: skipping nameserver 'ns0.openstack.codfw1dev.wikimediacloud.org' because it is a CNAME, while resolving '0-29.57.15.185.in-addr.arpa/NS'
The cause is a relatively archaic DNS rule that hostnames references in NS entries should point directly to A/AAAA records, and not CNAMEs. Many resolvers ignore this, and it works, but Bind is not uncommon and the fact it cannot resolve these names may be a concern.

This is good to know! I only recently changed those to CNAMEs, so I'll switch them back when I'm back in the office.

In T315955#8188684, @Andrew wrote:

This is good to know! I only recently changed those to CNAMEs, so I'll switch them back when I'm back in the office.

https://gerrit.wikimedia.org/r/c/operations/dns/+/827446/

Change 827446 had a related patch set uploaded (by Andrew Bogott; author: Majavah):

[operations/dns@master] wikimediacloud.org: do not use CNAMEs for nsX addresses

https://gerrit.wikimedia.org/r/827446

Change 827446 merged by Andrew Bogott:

[operations/dns@master] wikimediacloud.org: do not use CNAMEs for nsX addresses

https://gerrit.wikimedia.org/r/827446

Andrew closed this task as Resolved.Aug 31 2022, 8:58 PM

Change 826803 merged by Cathal Mooney:

[operations/dns@master] Sub-delegation of reverse DNS entries for 185.15.57.16/29 to WMCS

https://gerrit.wikimedia.org/r/826803

Just to confirm the change from CNAME back to A records has worked, my BIND server at home is able to resolve WMCS names again.

In terms of the reverse entries for 185.15.57.16/29 I've merged the patch to delegate these on our authdns:

cathal@officepc:~$ dig +noall +authority NS 16-29.57.15.185.in-addr.arpa. @ns0.wikimedia.org
16-29.57.15.185.in-addr.arpa. 3600 IN   NS      ns0.openstack.codfw1dev.wikimediacloud.org.
16-29.57.15.185.in-addr.arpa. 3600 IN   NS      ns1.openstack.codfw1dev.wikimediacloud.org.

However the WMCS name servers return NXDOMAIN for the zone still:

cathal@officepc:~$ dig SOA 16-29.57.15.185.in-addr.arpa. @ns0.openstack.codfw1dev.wikimediacloud.org

; <<>> DiG 9.18.1-1ubuntu1.1-Ubuntu <<>> SOA 16-29.57.15.185.in-addr.arpa. @ns0.openstack.codfw1dev.wikimediacloud.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 52993

This is in contrast to the zone for 185.15.57.0/16:

cathal@officepc:~$ dig +noall +answer SOA 0-29.57.15.185.in-addr.arpa. @ns0.openstack.codfw1dev.wikimediacloud.org
0-29.57.15.185.in-addr.arpa. 3600 IN    SOA     ns1.openstack.codfw1dev.wikimediacloud.org. root.wmflabs.org. 1650969226 3547 600 86400 3600

If 16-29.57.15.185.in-addr.arpa. is configured on the WMCS NS servers similar to 0-29.57.15.185.in-addr.arpa. it should work.

Maintenance_bot removed a project: Patch-For-Review.Sep 7 2022, 11:31 AM

root@cloudcontrol2005-dev:~# dig +noall +answer SOA 16-29.57.15.185.in-addr.arpa.
16-29.57.15.185.in-addr.arpa. 120 IN	SOA	ns0.openstack.codfw1dev.wikimediacloud.org. root.wmflabs.org. 1662658704 3517 600 86400 3600

That's the last piece, right?

@Andrew yep that's what was needed from the zone side so looking good there.

It's not actually returning any data for specific IPs in the range though. Comparing, for instance, to the 185.15.57.0/29 we see CNAMEs for all 6 IPs (these come from the Wikimedia name servers), and then PTRs for some of the CNAMEs (these come from the 0-29.57.15.185.in-addr.arpa. zone on the WMCS name servers):

cathal@officepc:~$ for i in {0..7}; do dig +noall +answer -x 185.15.57.$i; done
0.57.15.185.in-addr.arpa. 3600  IN      CNAME   0.0-29.57.15.185.in-addr.arpa.
1.57.15.185.in-addr.arpa. 3600  IN      CNAME   1.0-29.57.15.185.in-addr.arpa.
1.0-29.57.15.185.in-addr.arpa. 3600 IN  PTR     nat.cloudgw.codfw1dev.wikimediacloud.org.
2.57.15.185.in-addr.arpa. 3600  IN      CNAME   2.0-29.57.15.185.in-addr.arpa.
2.0-29.57.15.185.in-addr.arpa. 3600 IN  PTR     bastion.bastioninfra-codfw1dev.codfw1dev.wmcloud.org.
3.57.15.185.in-addr.arpa. 3600  IN      CNAME   3.0-29.57.15.185.in-addr.arpa.
4.57.15.185.in-addr.arpa. 3600  IN      CNAME   4.0-29.57.15.185.in-addr.arpa.
5.57.15.185.in-addr.arpa. 3600  IN      CNAME   5.0-29.57.15.185.in-addr.arpa.
5.0-29.57.15.185.in-addr.arpa. 3600 IN  PTR     manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org.
6.57.15.185.in-addr.arpa. 3600  IN      CNAME   6.0-29.57.15.185.in-addr.arpa.
7.57.15.185.in-addr.arpa. 3600  IN      CNAME   7.0-29.57.15.185.in-addr.arpa.

Comparing that to the new range/zone there are no PTRs coming back, so I guess in terms of this original task we still aren't seeing any entry for 185.15.57.20. But I'm also aware maybe none of these are in use anymore:

cathal@officepc:~$ for i in {16..23}; do dig +noall +answer -x 185.15.57.$i; done
16.57.15.185.in-addr.arpa. 3600 IN      CNAME   16.16-29.57.15.185.in-addr.arpa.
17.57.15.185.in-addr.arpa. 3600 IN      CNAME   17.16-29.57.15.185.in-addr.arpa.
18.57.15.185.in-addr.arpa. 3600 IN      CNAME   18.16-29.57.15.185.in-addr.arpa.
19.57.15.185.in-addr.arpa. 3600 IN      CNAME   19.16-29.57.15.185.in-addr.arpa.
20.57.15.185.in-addr.arpa. 3600 IN      CNAME   20.16-29.57.15.185.in-addr.arpa.
21.57.15.185.in-addr.arpa. 3600 IN      CNAME   21.16-29.57.15.185.in-addr.arpa.
22.57.15.185.in-addr.arpa. 3600 IN      CNAME   22.16-29.57.15.185.in-addr.arpa.
23.57.15.185.in-addr.arpa. 3600 IN      CNAME   23.16-29.57.15.185.in-addr.arpa.

Yep, we're using those IPs for rapid tests so most of the time they're unallocated.

Ok cool well we can close this in that case I think. Cheers.

Undocumented IP on WMCS networkClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Undocumented IP on WMCS network
Closed, ResolvedPublic
Actions