DNS for our nova-network public range is delegated to the nameservers managed by designate:
templates/155.80.208.in-addr.arpa
; 208.80.155.128/25 Eqiad Labs virtualization subnet
; Delegate 208.80.155.128 - 208.80.155.255 to labs-ns*
128-25 IN NS labs-ns0.wikimedia.org.
IN NS labs-ns1.wikimedia.org.We need to do the same for the range assigned in T193496 i.e. 185.15.56.0/24
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T53494 Use Beta cluster as a true canary for code deployments (epic) | |||
| Declined | None | T87220 Minimize infrastructure differences between Beta Cluster and production | |||
| Open | None | T196662 Set up LVS in beta like prod | |||
| Resolved | bd808 | T166396 Program 1 Outcome 4: VPS hosting | |||
| Resolved | None | T167293 Nova-network to Neutron migration | |||
| Resolved | Andrew | T199374 Delegate public IP range for Eqiad1-r OpenStack deployment to designate (neutron) |
So someone with novaadmin will need to create a 56.15.185.in-addr.arpa designate zone in the wmflabsdotorg project.
Then we'll need to do some refactoring around the floating IP DNS update scripting to make it work on a second zone that doesn't need RFC 2317 trickery
Change 445303 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] Delegate 185.15.56.0/24 to labs-ns0/ns1
Change 445310 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] openstack eqiad1: Run dns_floating_ip_updater
Change 445310 merged by Andrew Bogott:
[operations/puppet@production] openstack eqiad1: Run dns_floating_ip_updater
I merged https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/445310/ and the entries created look right to me. I can't actually dig -x yet though, presumable because we still need the delegation patch.
Change 445303 merged by Andrew Bogott:
[operations/dns@master] Delegate 185.15.56.0/24 to labs-ns0/ns1
As long as you point it at the right DNS server you should be fine regardless of delegations, but:
alex@alex-laptop:~$ dig eqiad1.bastion.wmflabs.org +short 185.15.56.13 alex@alex-laptop:~$ dig -x 185.15.56.13 @labs-ns0.wikimedia.org +short alex@alex-laptop:~$
looking at horizon, the wmflabsdotorg project's 56.15.185.in-addr.arpa. zone clearly has a 13.56.15.185.in-addr.arpa. PTR record with the value eqiad1.bastion.wmflabs.org. (so my script appears to be working) - but when I query labs-ns0 or labs-ns1 for it, the record is nowhere to be seen?
alex@alex-laptop:~$ dig 13.56.15.185.in-addr.arpa PTR @labs-ns0.wikimedia.org ; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> 13.56.15.185.in-addr.arpa PTR @labs-ns0.wikimedia.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42930 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 2800 ;; QUESTION SECTION: ;13.56.15.185.in-addr.arpa. IN PTR ;; Query time: 162 msec ;; SERVER: 208.80.155.117#53(208.80.155.117) ;; WHEN: Fri Aug 31 23:15:51 BST 2018 ;; MSG SIZE rcvd: 54
labs-ns* responses look fine now though. @Andrew might this be to do with the eqiad <-> eqiad1-r designate change you made today?
There is a problem with the delegation though, prod isn't returning the NS record into labs where I'd expect, instead we get the SOA?
alex@alex-laptop:~$ dig 13.56.15.185.in-addr.arpa PTR +trace @8.8.8.8 [snip] 56.15.185.in-addr.arpa. 172800 IN NS ns0.wikimedia.org. 56.15.185.in-addr.arpa. 172800 IN NS ns1.wikimedia.org. 56.15.185.in-addr.arpa. 172800 IN NS ns2.wikimedia.org. 56.15.185.in-addr.arpa. 3600 IN NSEC 57.15.185.in-addr.arpa. NS RRSIG NSEC 56.15.185.in-addr.arpa. 3600 IN RRSIG NSEC 8 5 3600 20181002134304 20180902124304 64230 185.in-addr.arpa. cPGxkgvEanbIEsvtoALMTjLa6gqB/QluqS6e/glE+k0zE29reh9S8fxC MKW2EpeDDsWgMtMqRcAqslQBl2ZmcJLleMqlYM2NPioeLGivNKQCtxxD m3ZHFoZPnp+J6HMLuDczMihrQm6/mkvoaGZJNzbj370srhXjlvAD3Qxj ims= ;; Received 369 bytes from 199.212.0.53#53(tinnie.arin.net) in 82 ms 56.15.185.in-addr.arpa. 3600 IN SOA labs-ns0.wikimedia.org. hostmaster.wikimedia.org. 2018083120 43200 7200 1209600 3600 ;; Received 123 bytes from 208.80.154.238#53(ns0.wikimedia.org) in 173 ms alex@alex-laptop:~$ dig 13.56.15.185.in-addr.arpa PTR @labs-ns0.wikimedia.org +short eqiad1.bastion.wmflabs.org.
So I wonder if that operations/dns.git commit is correct. It might be easier if we just have them point reverse DNS queries straight at the labs nameservers instead of going via prod gdnsd. That wasn't possible on the old setup as our range was a /25 (208.80.155.128/25), and the other /25 within that /24 (208.80.155.0/25) was prod stuff, but we can do it here as the entire /24 (182.15.56.0/24) is reserved for labs.
After https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/457460/ and fixing a silly typo, I now think that the ptr records should be updating correctly and promptly.
I don't really know what the NS record should look like... maybe @BBlack has thoughts?
I wonder if gdnsd doesn't like the NS record out to another server at the zone apex like that. Could create an NS record for each of 0-255 I guess.
I spoke with Brandon Black in -traffic just now, basically you can't delegate a full zone like this (nor in several other ways I suggested) - and changing RIPE to point directly at labs-ns* might be complicated by the fact that WMF owns the larger /22 rather than just this /24
(edit: he wrote the best way, if supported by RIPE, would be to ask RIPE to change the delegation, but only for this 1x /24 (which is part of a contiguous /22 they gave us, so I'm not sure of that's possible))
Here's what we can do, to set up in a similar way to the old 208.80.155.128/25 (I had been thinking we could avoid this but apparently not):
(designate zones should/can be found under the wmflabsdotorg tenant)
I just edited the 56.15.185.in-addr.arpa object in the RIPE database to point nameservers directly to labs-ns0/1. This should work now, no need for classful classless delegation :)
Change 462574 had a related patch set uploaded (by Faidon Liambotis; owner: Faidon Liambotis):
[operations/dns@master] Remove 56.15.185.in-addr.arpa, moved to labs-ns0/1
Thanks @faidon
alex@alex-laptop:~$ host eqiad1.bastion.wmflabs.org eqiad1.bastion.wmflabs.org has address 185.15.56.13 alex@alex-laptop:~$ host 185.15.56.13 13.56.15.185.in-addr.arpa domain name pointer eqiad1.bastion.wmflabs.org.
alex@alex-laptop:~$ host smtp-out1001.project-smtp.wmflabs.org smtp-out1001.project-smtp.wmflabs.org has address 185.15.56.15 alex@alex-laptop:~$ host 185.15.56.15 15.56.15.185.in-addr.arpa domain name pointer smtp-out1001.project-smtp.wmflabs.org.
Change 462574 merged by Faidon Liambotis:
[operations/dns@master] Remove 56.15.185.in-addr.arpa, moved to labs-ns0/1
Note that there is the RIPE's TTL at play here, which is set to 172800 (2 days), so I'd wait for that before I start using these records.