We've had reports that certain internet connected systems are failing to resolve hostnames under toolforge.org. This follows some changes WMCS have made today which have changed the IP address for //ns1.openstack.eqiad1.wikimediacloud.org.//
The wikimediacloud.org domain is hosted by ns[0-2].wikimedia.org, and this is working normally. It is returning the following two A records right now:
```
cathal@officepc:~$ dig +noall +answer A ns0.openstack.eqiad1.wikimediacloud.org. @ns0.wikimedia.org
ns0.openstack.eqiad1.wikimediacloud.org. 300 IN A 208.80.154.148
```
cathal@officepc:~$ dig +noall +answer A ns1.openstack.eqiad1.wikimediacloud.org. @ns0.wikimedia.org
ns1.openstack.eqiad1.wikimediacloud.org. 3600 IN A 185.15.56.163
```
The first is a manual record directly in the zone file, the second is a Netbox-generated record that's included in it (distinction irrelevant tbh).
I see if I query for the toolforge.org NS records from any of the .ORG TLDs they return the two old A records for these hostnames in the 'additional' section:
```
cathal@officepc:~$ dig +nsid NS toolforge.org @b2.org.afilias-nst.org.
; <<>> DiG 9.18.12-0ubuntu0.22.04.2-Ubuntu <<>> +nsid NS toolforge.org @b2.org.afilias-nst.org.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45215
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; NSID: 4c 48 52 35 ("LHR5")
;; QUESTION SECTION:
;toolforge.org. IN NS
;; AUTHORITY SECTION:
toolforge.org. 3600 IN NS ns0.openstack.eqiad1.wikimediacloud.org.
toolforge.org. 3600 IN NS ns1.openstack.eqiad1.wikimediacloud.org.
;; ADDITIONAL SECTION:
ns1.openstack.eqiad1.wikimediacloud.org. 3600 IN A 208.80.154.11
ns0.openstack.eqiad1.wikimediacloud.org. 3600 IN A 208.80.154.135
;; Query time: 40 msec
;; SERVER: 2001:500:48::1#53(b2.org.afilias-nst.org.) (UDP)
;; WHEN: Tue Sep 12 19:04:51 IST 2023
;; MSG SIZE rcvd: 153
```
While there is no circular dependency here (the name servers hostnames for toolforge.org are not under toolforge.org), it seems to be that the ORG TLDs may be hard-coded with these A records / IPs. And that may be what is causing the problems we seem to be having.