Page MenuHomePhabricator

Delegate public IP range for Eqiad1-r OpenStack deployment to designate (neutron)
Closed, ResolvedPublic

Description

DNS for our nova-network public range is delegated to the nameservers managed by designate:

templates/155.80.208.in-addr.arpa

; 208.80.155.128/25 Eqiad Labs virtualization subnet
; Delegate 208.80.155.128 - 208.80.155.255 to labs-ns*

128-25 IN NS labs-ns0.wikimedia.org.
       IN NS labs-ns1.wikimedia.org.

We need to do the same for the range assigned in T193496 i.e. 185.15.56.0/24

Event Timeline

chasemp triaged this task as Normal priority.Jul 11 2018, 9:05 PM
chasemp created this task.
chasemp updated the task description. (Show Details)
chasemp updated the task description. (Show Details)

So someone with novaadmin will need to create a 56.15.185.in-addr.arpa designate zone in the wmflabsdotorg project.
Then we'll need to do some refactoring around the floating IP DNS update scripting to make it work on a second zone that doesn't need RFC 2317 trickery

Change 445303 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] Delegate 185.15.56.0/24 to labs-ns0/ns1

https://gerrit.wikimedia.org/r/445303

Change 445310 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] openstack eqiad1: Run dns_floating_ip_updater

https://gerrit.wikimedia.org/r/445310

Change 445310 merged by Andrew Bogott:
[operations/puppet@production] openstack eqiad1: Run dns_floating_ip_updater

https://gerrit.wikimedia.org/r/445310

Andrew claimed this task.Aug 31 2018, 8:35 PM

I merged https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/445310/ and the entries created look right to me. I can't actually dig -x yet though, presumable because we still need the delegation patch.

Change 445303 merged by Andrew Bogott:
[operations/dns@master] Delegate 185.15.56.0/24 to labs-ns0/ns1

https://gerrit.wikimedia.org/r/445303

Krenair added a comment.EditedAug 31 2018, 10:03 PM

I merged https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/445310/ and the entries created look right to me. I can't actually dig -x yet though, presumable because we still need the delegation patch.

As long as you point it at the right DNS server you should be fine regardless of delegations, but:

alex@alex-laptop:~$ dig eqiad1.bastion.wmflabs.org +short
185.15.56.13
alex@alex-laptop:~$ dig -x 185.15.56.13 @labs-ns0.wikimedia.org +short
alex@alex-laptop:~$
Krenair added a comment.EditedAug 31 2018, 10:17 PM

looking at horizon, the wmflabsdotorg project's 56.15.185.in-addr.arpa. zone clearly has a 13.56.15.185.in-addr.arpa. PTR record with the value eqiad1.bastion.wmflabs.org. (so my script appears to be working) - but when I query labs-ns0 or labs-ns1 for it, the record is nowhere to be seen?

alex@alex-laptop:~$ dig 13.56.15.185.in-addr.arpa PTR @labs-ns0.wikimedia.org

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> 13.56.15.185.in-addr.arpa PTR @labs-ns0.wikimedia.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42930
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 2800
;; QUESTION SECTION:
;13.56.15.185.in-addr.arpa.	IN	PTR

;; Query time: 162 msec
;; SERVER: 208.80.155.117#53(208.80.155.117)
;; WHEN: Fri Aug 31 23:15:51 BST 2018
;; MSG SIZE  rcvd: 54

Krenair added a comment.EditedSep 2 2018, 4:27 PM

labs-ns* responses look fine now though. @Andrew might this be to do with the eqiad <-> eqiad1-r designate change you made today?

Krenair added a comment.EditedSep 2 2018, 4:37 PM

There is a problem with the delegation though, prod isn't returning the NS record into labs where I'd expect, instead we get the SOA?

alex@alex-laptop:~$ dig 13.56.15.185.in-addr.arpa PTR +trace @8.8.8.8

[snip]

56.15.185.in-addr.arpa.	172800	IN	NS	ns0.wikimedia.org.
56.15.185.in-addr.arpa.	172800	IN	NS	ns1.wikimedia.org.
56.15.185.in-addr.arpa.	172800	IN	NS	ns2.wikimedia.org.
56.15.185.in-addr.arpa.	3600	IN	NSEC	57.15.185.in-addr.arpa. NS RRSIG NSEC
56.15.185.in-addr.arpa.	3600	IN	RRSIG	NSEC 8 5 3600 20181002134304 20180902124304 64230 185.in-addr.arpa. cPGxkgvEanbIEsvtoALMTjLa6gqB/QluqS6e/glE+k0zE29reh9S8fxC MKW2EpeDDsWgMtMqRcAqslQBl2ZmcJLleMqlYM2NPioeLGivNKQCtxxD m3ZHFoZPnp+J6HMLuDczMihrQm6/mkvoaGZJNzbj370srhXjlvAD3Qxj ims=
;; Received 369 bytes from 199.212.0.53#53(tinnie.arin.net) in 82 ms

56.15.185.in-addr.arpa.	3600	IN	SOA	labs-ns0.wikimedia.org. hostmaster.wikimedia.org. 2018083120 43200 7200 1209600 3600
;; Received 123 bytes from 208.80.154.238#53(ns0.wikimedia.org) in 173 ms
alex@alex-laptop:~$ dig 13.56.15.185.in-addr.arpa PTR @labs-ns0.wikimedia.org +short
eqiad1.bastion.wmflabs.org.

So I wonder if that operations/dns.git commit is correct. It might be easier if we just have them point reverse DNS queries straight at the labs nameservers instead of going via prod gdnsd. That wasn't possible on the old setup as our range was a /25 (208.80.155.128/25), and the other /25 within that /24 (208.80.155.0/25) was prod stuff, but we can do it here as the entire /24 (182.15.56.0/24) is reserved for labs.

Andrew added a subscriber: BBlack.Sep 3 2018, 3:19 PM

After https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/457460/ and fixing a silly typo, I now think that the ptr records should be updating correctly and promptly.

I don't really know what the NS record should look like... maybe @BBlack has thoughts?

I wonder if gdnsd doesn't like the NS record out to another server at the zone apex like that. Could create an NS record for each of 0-255 I guess.

Krenair added a comment.EditedSep 24 2018, 5:31 PM

I spoke with Brandon Black in -traffic just now, basically you can't delegate a full zone like this (nor in several other ways I suggested) - and changing RIPE to point directly at labs-ns* might be complicated by the fact that WMF owns the larger /22 rather than just this /24
(edit: he wrote the best way, if supported by RIPE, would be to ask RIPE to change the delegation, but only for this 1x /24 (which is part of a contiguous /22 they gave us, so I'm not sure of that's possible))
Here's what we can do, to set up in a similar way to the old 208.80.155.128/25 (I had been thinking we could avoid this but apparently not):

  1. create 0-24.56.15.185.in-addr.arpa. zone in designate
  2. change hieradata/eqiad/profile/openstack/eqiad1/designate.yaml: profile::openstack::eqiad1::designate::floating_ip_ptr_fqdn_replacement_pattern: '\1.0-24.56.15.185.in-addr.arpa.'
  3. let it run
  4. change templates/56.15.185.in-addr.arpa to be like https://gerrit.wikimedia.org/r/#/c/operations/dns/+/445303/2/templates/56.15.185.in-addr.arpa (edit: Also Brandon said maybe bump up the delegation + CNAME record TTLs all to 1D)
  5. test it all is working
  6. delete old 56.15.185.in-addr.arpa zone from designate

(designate zones should/can be found under the wmflabsdotorg tenant)

herron added a subscriber: herron.Sep 24 2018, 7:36 PM
faidon added a subscriber: faidon.Sep 24 2018, 9:00 PM

I just edited the 56.15.185.in-addr.arpa object in the RIPE database to point nameservers directly to labs-ns0/1. This should work now, no need for classful classless delegation :)

Change 462574 had a related patch set uploaded (by Faidon Liambotis; owner: Faidon Liambotis):
[operations/dns@master] Remove 56.15.185.in-addr.arpa, moved to labs-ns0/1

https://gerrit.wikimedia.org/r/462574

Krenair closed this task as Resolved.Sep 24 2018, 9:08 PM

Thanks @faidon

alex@alex-laptop:~$ host eqiad1.bastion.wmflabs.org
eqiad1.bastion.wmflabs.org has address 185.15.56.13
alex@alex-laptop:~$ host 185.15.56.13
13.56.15.185.in-addr.arpa domain name pointer eqiad1.bastion.wmflabs.org.
alex@alex-laptop:~$ host smtp-out1001.project-smtp.wmflabs.org
smtp-out1001.project-smtp.wmflabs.org has address 185.15.56.15
alex@alex-laptop:~$ host 185.15.56.15
15.56.15.185.in-addr.arpa domain name pointer smtp-out1001.project-smtp.wmflabs.org.

Change 462574 merged by Faidon Liambotis:
[operations/dns@master] Remove 56.15.185.in-addr.arpa, moved to labs-ns0/1

https://gerrit.wikimedia.org/r/462574

Note that there is the RIPE's TTL at play here, which is set to 172800 (2 days), so I'd wait for that before I start using these records.