In discussion with @cmooney, we will be revisiting this task again when Traffic does some other authdns-related work, so removing it from the Traffic-Icebox.

I note that a current draft in the IETF DNSOPS Working Group, aimed to replace RFC3901, draft-momoka-dnsop-3901bis-03 states:

Every authoritative DNS zone SHOULD be served by at least one IPv6-reachable authoritative name server to maintain name space continuity. The delegation configuration (Resolution of the parent, resolution of out-of-bailiwick names, GLUE) MUST not rely on IPv4 connectivity being available.

Hi, Is there any update on this? I see some *.wikimedia.org have ipv6 addresses which is useless if the host is ipv6 only. Since they can't query the dns as it's ipv4 only.

Yes, thanks for the ping @Paladox. We should most certainly pick this up again. @BBlack: any fresh 2026 thoughts? You listed some concerns above but some of them don't apply anymore -- should we do a quick sync here to understand if you have any lingering concerns?

@cmooney and I can take care of the deployment (there was a time when our v6 bird unicast setup wasn't tested, but that's also not true anymore so that's not a blocker).

I should mention that ns[01] v6 will be unicast, like v4, and ns2 will be anycast v6, just like the v4 one. But these are minor operational details, the real question is if we are ready to do this from an authdns point of view.

We have to take this plunge someday, and that someday probably should've been years ago, just too many other pressing things to focus on for anyone to remember to come back here and look! A few notes:

• Obviously, it would make sense to take this on in phases: one nsX at a time, space it out by long enough (a week or two) to validate all the things and have time for any non-obvious failure reports to percolate up to us.
• Currently, our upstream (in the DNS hierarchy) NS record TTLs are all 3600. This actually seems quite a bit shorter than it should be, to me. I don't recall the history on this, but I would've expected something more like 86400, which is what we set in our local zonefiles (the local records in our zonefiles matter far less to resolvers than the upstream ones in the .org servers, but it all still matters!).
• My bikeshed on the TTL issues would be: I would drop our local TTLs on these to 3600 like upstream ahead of these changes, just because it makes rollback a bit less slow (assuming the registrar has fast processes), and keep A and AAAA having matching TTLs at all times, and bump it all (.org via the registrar, and our zonefiles) to 86400 after we're done with the IPv6 change.

[In fact, on that point, I'd note a quick survey of a handful of other major sites on the Internet shows a common pattern of 2 days for the NS records and 2-4 days on the matching address records. In any case, getting us to 1 day everywhere consistently would be a good start!]

In T81605#11517321, @BBlack wrote:

We have to take this plunge someday, and that someday probably should've been years ago, just too many other pressing things to focus on for anyone to remember to come back here and look! A few notes:

• Obviously, it would make sense to take this on in phases: one nsX at a time, space it out by long enough (a week or two) to validate all the things and have time for any non-obvious failure reports to percolate up to us.

Makes sense -- https://w.wiki/HSCH indicates per wmf_netflow that the anycast ns2 gets the most traffic, followed by ns0 and ns1. So we can pick ns1 in codfw for a safe, test rollout.

• Currently, our upstream (in the DNS hierarchy) NS record TTLs are all 3600. This actually seems quite a bit shorter than it should be, to me. I don't recall the history on this, but I would've expected something more like 86400, which is what we set in our local zonefiles (the local records in our zonefiles matter far less to resolvers than the upstream ones in the .org servers, but it all still matters!).

Yeah that's certainly interesting; I wasn't aware of that so we can patch it up and fix it. Looking at Markmonitor, I am assuming this is somewhat of a default on their end because we are not setting one explicitly and so we have just carried this over?

• My bikeshed on the TTL issues would be: I would drop our local TTLs on these to 3600 like upstream ahead of these changes, just because it makes rollback a bit less slow (assuming the registrar has fast processes), and keep A and AAAA having matching TTLs at all times, and bump it all (.org via the registrar, and our zonefiles) to 86400 after we're done with the IPv6 change.

Sounds good, thanks. While we do that, let us know if you think of something else before we roll this out :)

Our glue records also have a disparity.

dig wikimedia.org NS +trace +additional
ns2.wikimedia.org.	3600	IN	A	198.35.27.27
ns1.wikimedia.org.	3600	IN	A	208.80.153.231
ns0.wikimedia.org.	3600	IN	A	208.80.154.238

whereas, we set them to:

ns0         1D  IN A    208.80.154.238
ns1         1D  IN A    208.80.153.231
ns2         1D  IN A    198.35.27.27 ; anycasted authdns

Change #1226904 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] wikimedia/wikipedia.org: match TTLs for NS and glue records

https://gerrit.wikimedia.org/r/1226904

@cmooney: Any picks for your favourite v6 address for ns1? I was thinking of allocating 2620:0:860:ed1a::4/128 under LVS service IPs 2620:0:860:ed1a::/64, since unfortunately that is where have put the v4s, but I also don't want to carry that mistake forward, so deferring to you on that.

Change #1226928 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] dnsbox: codfw: advertise ns1 IPv6

https://gerrit.wikimedia.org/r/1226928

In T81605#11522551, @ssingh wrote:

@cmooney: Any picks for your favourite v6 address for ns1? I was thinking of allocating 2620:0:860:ed1a::4/128 under LVS service IPs 2620:0:860:ed1a::/64, since unfortunately that is where have put the v4s, but I also don't want to carry that mistake forward, so deferring to you on that.

Yeah it's not behind the LVS so I think somewhat confusing to allocate it an address from that range.

I'd be tempted to assign 2620:0:860:53::/64 from the codfw public subnets /56 for this. And then use 2620:0:860:53::/128 as the ns1 address?

We could possibly follow the same approach for ns0? Or is there an existing plan for eqiad?

In T81605#11518553, @ssingh wrote:

Our glue records also have a disparity.

I was interested to know what effect this would have. One data-point for Bind (at least my local instance) is that it is caching the TTL that comes back from our servers, not the glue records in the ORG zone:

cathal@officepc:~$ dig A ns0.wikimedia.org @192.168.240.1

; <<>> DiG 9.20.15-1~deb13u1-Debian <<>> A ns0.wikimedia.org @192.168.240.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55864
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1400
; COOKIE: 6d3761bd9ecfb45301000000696a50bef494110bbf7eb76b (good)
;; QUESTION SECTION:
;ns0.wikimedia.org.		IN	A

;; ANSWER SECTION:
ns0.wikimedia.org.	30297	IN	A	208.80.154.238

In T81605#11529119, @cmooney wrote:

In T81605#11522551, @ssingh wrote:

@cmooney: Any picks for your favourite v6 address for ns1? I was thinking of allocating 2620:0:860:ed1a::4/128 under LVS service IPs 2620:0:860:ed1a::/64, since unfortunately that is where have put the v4s, but I also don't want to carry that mistake forward, so deferring to you on that.

Yeah it's not behind the LVS so I think somewhat confusing to allocate it an address from that range.

I'd be tempted to assign 2620:0:860:53::/64 from the codfw public subnets /56 for this. And then use 2620:0:860:53::/128 as the ns1 address?

We could possibly follow the same approach for ns0? Or is there an existing plan for eqiad?

Thanks! The plan is to the same for eqiad and then find a range for the anycast of ns2. Any thoughts on the last one (anycast range)? We will do that at the very end given ns2 sees the most traffic but checking.

In T81605#11529426, @ssingh wrote:

Thanks! The plan is to the same for eqiad

Ok I've reserved those two ranges/IPs in Netbox now.

Any thoughts on the last one (anycast range)?

We'll want a new /48 separate prefix to announce into the DFZ for it I assume? We can possibly allocate 2a02:ec80:53::/40 for it in Netbox (better to only carve up the block at that size to keep things consistent), announce 2a02:ec80:53::/48 to the internet, and then use 2a02:ec80:53::/128 as the IP?

In T81605#11529540, @cmooney wrote:

In T81605#11529426, @ssingh wrote:

Thanks! The plan is to the same for eqiad

Ok I've reserved those two ranges/IPs in Netbox now.

Many thanks!

Any thoughts on the last one (anycast range)?

We'll want a new /48 separate prefix to announce into the DFZ for it I assume? We can possibly allocate 2a02:ec80:53::/40 for it in Netbox (better to only carve up the block at that size to keep things consistent), announce 2a02:ec80:53::/48 to the internet, and then use 2a02:ec80:53::/128 as the IP?

That seems to be what we are doing for Wikimedia DNS (and what we did), so I guess it makes sense -- but I am leaving that to your expertise :)

(2001:67c:930::/48 and then 2001:67c:930::1/128 for the Wikimedia DNS v6 anycast.)

overall lgtm

Using a full /64 unicast 2620:0:860:53::/64 for a single service looks a bit weird, but as it's something critical like AuthDNS it doesn't shock me too much. The alternative is to allocate a codfw /64 for any kind of "bird" services, one of them being authdns. Like we have a LVS range, which contains multiple services.

We'll want a new /48 separate prefix to announce into the DFZ for it I assume?

It's not strictly needed (especially if we internally route it in case of local failure), but doesn't hut to add one more entry in the DFZ to give us more control on the announcements.

with a slight preference to use 2a02:ec80:53::1/128 rather than 2a02:ec80:53::/128 as IPs to not freak people out.

In T81605#11530021, @ayounsi wrote:

overall lgtm

Using a full /64 unicast 2620:0:860:53::/64 for a single service looks a bit weird, but as it's something critical like AuthDNS it doesn't shock me too much. The alternative is to allocate a codfw /64 for any kind of "bird" services, one of them being authdns. Like we have a LVS range, which contains multiple services.

Not that it makes it any better but we are "wasting" IPv4s as well for the ns2 and the Wikimedia DNS anycast ranges, and the rationale we applied at that time as well was to keep them distinct. So I am guessing the IPv6 addresses don't have the same shortage/resource constraints and therefore we can continue using the full /64 unless there are other considerations from your end.

We'll want a new /48 separate prefix to announce into the DFZ for it I assume?

It's not strictly needed (especially if we internally route it in case of local failure), but doesn't hut to add one more entry in the DFZ to give us more control on the announcements.

with a slight preference to use 2a02:ec80:53::1/128 rather than 2a02:ec80:53::/128 as IPs to not freak people out.

I had that in mind as we do the same for text-lb, upload-lb, Wikimedia DNS but no strong preferences here, so leaving that you both. (FWIW, Cathal already assigned 2a02:ec80:53::/128).

In T81605#11529143, @cmooney wrote:

In T81605#11518553, @ssingh wrote:

Our glue records also have a disparity.

I was interested to know what effect this would have. One data-point for Bind (at least my local instance) is that it is caching the TTL that comes back from our servers, not the glue records in the ORG zone:

cathal@officepc:~$ dig A ns0.wikimedia.org @192.168.240.1

[...]

Except almost nobody but engineers are going to directly query that record. Most caches will learn and re-learn it as they traverse the delegation from the root of the DNS, and thus if the end-users are querying say en.wikipedia.org, the cache will learn it from the glue records at the Afilias org-level authservers. But sometimes they may learn it from us, if they happen to first traverse one of our non-.org domains that uses our .org NS records.

In T81605#11530209, @BBlack wrote:

Except almost nobody but engineers are going to directly query that record. Most caches will learn and re-learn it as they traverse the delegation from the root of the DNS, and thus if the end-users are querying say en.wikipedia.org, the cache will learn it from the glue records at the Afilias org-level authservers. But sometimes they may learn it from us, if they happen to first traverse one of our non-.org domains that uses our .org NS records.

That doesn't seem to be happening for me. I did a pcap to see what was going on, and made a query for 'en.wikipedia.org' with dig.

{F71570195}

It seems after Bind gets the NS query response from the .org servers, including the glue 'A' entries for our authdns (packet 23), it immediately makes follow-up 'A' queries to our authdns for those same names, using the IPs from glue (packets 24-29). It's the responses to those, from our authdns with the TTL we set, that Bind caches.

In T81605#11530397, @cmooney wrote:

As a further test I wiped my cache, started a packet capture and did a dig for 'en.wikimedia.org'.

did you intend to use a non-canonical domain here? pedia vs media.

In T81605#11530397, @cmooney wrote:

But it seems Bind does not cache the glue records / additional that comes back from the .org authdns. At least for any length of time. It immediately makes separate A record queries for the nameservers .org returns, sent to the IPs from the glue records. It's the responses from those direct queries to us that it caches, with a TTL of 86400.

That all makes sense, and once they're loaded by that direct query, it would probably even use them in place of any missing "glue" (e.g. for wikipedia.com lookups that return NS delegations without addresses glued). But still, "nobody" is directly querying the nsX A-records in the normal mass case in public resolvers. I would expect it does cache the glue (attached to the NS) for an hour in the true delegation case.

In T81605#11530539, @jeremyb wrote:

did you intend to use a non-canonical domain here? pedia vs media.

Ah sorry that was a typo, corrected now. I looked up en.wikipedia.org (pcap doesn't lie at least).

In T81605#11530621, @BBlack wrote:

That all makes sense, and once they're loaded by that direct query, it would probably even use them in place of any missing "glue"

Yeah those subsequent A record queries - when it already has the glue records from .org servers - were a surprise. I wonder if that's common to other recursors or only Bind? I guess doesn't really matter as mostly they should be identical.

Change #1228518 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/software/homer/deploy@master] plugins/wmf-netbox: remove ipv4 only for DNS hosts BGP

https://gerrit.wikimedia.org/r/1228518

@cmooney: Per the discussion above with Arzhel, we think that 2a02:ec80:53::1/128 is better for readability and consistency with other v6 records, than the current 2a02:ec80:53::/128. Any thoughts on that? If this makes sense to you as well, please feel free to re-assign in Netbox (or ask me to do it :) and we can move ahead with the other stuff. Thanks!

Change #1228576 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Add ns2.wikimedia.org anycast block to anycast config

https://gerrit.wikimedia.org/r/1228576

In T81605#11534944, @ssingh wrote:

@cmooney: Per the discussion above with Arzhel, we think that 2a02:ec80:53::1/128 is better for readability and consistency with other v6 records, than the current 2a02:ec80:53::/128. Any thoughts on that? If this makes sense to you as well, please feel free to re-assign in Netbox (or ask me to do it :) and we can move ahead with the other stuff. Thanks!

Yeah that's fine. The anycast IP hadn't been assigned in Netbox, so I've done that now. I've also create the RIPE IRR records for the block and the required RPKI ROA object.

So to confirm the assigned addresses are:

2620:0:861:53::1    ns0.wikimedia.org
2620:0:860:53::1    ns1.wikimedia.org
2a02:ec80:53::1     ns2.wikimedia.org

Many thanks @cmooney 🙏! I will go ahead with 2620:0:860:53::1/128 for ns1 and update that everywhere in the current CRs.

As discussed I think a good way to bring this live might be:

1. Merge patch in puppet repo to make Bird announce the new IPs at all sites
2. Merge the patch to enable IPv6 peering to the dns-boxes in the homer-reploy repo
3. Release new homer version (plugin only release) to include this change
4. Merge the patch to accept the new anycast addresses over BGP from the authdns boxes (done)
5. Run Homer against all our core routers

At this point all the authdns boxes should be announcing the new IPs to our core routers. We should be receiving them and they should be pingable from the internet. We can check:

1. Internet reachability
2. BGP announcements of the new Anycast /48 is being done at all points and accepted by upstreams
3. Manual queries direct to the IPs are answered (with dig)

Once we are happy all is good with that we can start adjusting our DNS zones, publishing the new AAAA records one by one starting with codfw/ns1. We should aim to add the record to our own boxes first, but quickly thereafter update the 'glue' with the registrar too.

In T81605#11535037, @cmooney wrote:

As discussed I think a good way to bring this live might be:
[...]

Sounds like a plan and it makes sense -- we can test everything on our end before enabling the change selectively, starting with ns1. The updated design also helps us avoid the problem of having to silence any alerts, which may hide other problems and is perhaps not desirable in hindsight.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1226928 has been updated to effectively roll this out everywhere on the DNS hosts.

We also have an additional layer of control on the DNS hosts, though the current design makes it slightly more coupled than desired. Essentially,

sukhe@puppetserver1001:~$ sudo confctl select 'name=dns2004.wikimedia.org' get
{"dns2004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=codfw,cluster=dnsbox,service=ntp-a"}
{"dns2004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=codfw,cluster=dnsbox,service=recdns"}
{"dns2004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=codfw,cluster=dnsbox,service=authdns-ns1"}
{"dns2004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=codfw,cluster=dnsbox,service=authdns-ns2"}
{"dns2004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=codfw,cluster=dnsbox,service=authdns-update"}

This means that we can theoretically depool authdns-ns1 but that means a depool of both v4 and v6. In some ways, this is not ideal. But then, I can't think of a reason on why we would like to keep one pooled but not the other, and perhaps there is value in having one service for both the addresses. But this is something that is up for discussion and if required, I can update the various bits to have two services per authdns-nsX, authdns-ns1-v4 and authdns-ns1-v6.

That's some extra work so perhaps for later but I just wanted to mention that in case someone else has thoughts.

A maybe safer alternative is to first enable IPv6 BGP peering between the network and all the dnsbox with profile::bird::do_ipv6: true (and the Homer patches). BGP over v6 will be established, but the dnsbox won't be advertising any prefix.
Then if all good merge a patch to make bird/anycast-hc advertise the new IPs.

Change #1228576 merged by jenkins-bot:

[operations/homer/public@master] Add config for authdns IPv6 public IPs

https://gerrit.wikimedia.org/r/1228576

In T81605#11535605, @ayounsi wrote:

A maybe safer alternative is to first enable IPv6 BGP peering between the network and all the dnsbox with profile::bird::do_ipv6: true (and the Homer patches). BGP over v6 will be established, but the dnsbox won't be advertising any prefix.
Then if all good merge a patch to make bird/anycast-hc advertise the new IPs.

Cool. I didn't know if the Bird role would support that (i.e. enable an IPv6 BGP peering without announcing any IPs).

Ultimately I think we are safe to set up the routing step-by-step or in one big bang, it's not until we publish the AAAA records that there is potential to disrupt anything. So I'll leave to @ssingh and Traffic team to make the call on how to make the Bird changes.

Yeah, unless we update our own zone files but more specifically, Markmonitor, nothing really changes so we can just go ahead with the approach @cmooney suggested and enable it everywhere so that we can test things, and then turn it on when we are ready by publishing the updated glue record.

What @ayounsi is suggesting will of course also work -- and that's the same thing when a service is depooled, bird is set up but doesn't announce the IP -- but I really don't think it's required.

Change #1226904 merged by Ssingh:

[operations/dns@master] wikimedia/wikipedia.org: match TTLs for NS and glue records

https://gerrit.wikimedia.org/r/1226904

There is another layer of complexity we need to be aware of. Essentially, authdns_addrs in hieradata/common.yaml has the list of v4 authdns IP records, and now will have the v6 ones. In the above Puppet/bird patch, I thought we could simply skip adding the v6 IPs there till we have everything running and I assumed that only dictates the configuration for haproxy for DNS over TLS on port 853. I was wrong -- and I only realized this after looking at the PCC output -- since that particular key sets up not only the DoTLS bits but also the gdnsd bits, including the listen addresses and firewall rules; that's the canonical list of IPs for the auth servers.

See https://puppet-compiler.wmflabs.org/output/1226928/7913/dns2004.wikimedia.org/index.html (authdns_addr set for the v6 bits) and compare it with https://puppet-compiler.wmflabs.org/output/1226928/7918/dns1004.wikimedia.org/index.html (authdns_addrs not set).

This is not a big deal since we are not actually publishing these records, so it doesn't matter if haproxy is listening on 853/v6 for DoTLS but we need to be aware of this.

Change #1228518 merged by Cathal Mooney:

[operations/software/homer/deploy@master] plugins/wmf-netbox: remove ipv4 only for DNS hosts BGP

https://gerrit.wikimedia.org/r/1228518

Mentioned in SAL (#wikimedia-operations) [2026-01-21T17:37:52Z] <sukhe> sudo cumin "A:dnsbox" "disable-puppet 'merging CR 1226928'": T81605

Change #1226928 merged by Ssingh:

[operations/puppet@production] dnsbox: advertise ns[0-2] IPv6

https://gerrit.wikimedia.org/r/1226928

Mentioned in SAL (#wikimedia-operations) [2026-01-21T19:30:02Z] <sukhe> re-enable puppet on A:dnsbox: T81605

Just want to add my two cents on the problem we hit trying to make the IPv6 IPs live.

• I personally think it's cleaner if the dns servers only have configured, and only listen on, the IPs that are actually routing to them
  • I reckon in a disaster it's as easy (and as delicate) to modify the dnsbox's at a site to listen on/announce extra IPs as to route them manually on the network
• Ultimately netops have no preference
  • We only care that the dns boxes announce the correct set of IPs at any location and answer queries sent to them
  • So whatever way traffic want to handle this is fine, we can either stop configuring core site IPs on the dns servers at POPs, or continue to do that but adjust the automation so it sets the IPv6 ones up with the correct netmask

Change #1230351 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] dnsbox: advertise ns[0-2] IPv6

https://gerrit.wikimedia.org/r/1230351

https://puppet-compiler.wmflabs.org/output/1230351/7941/

Interface::Ip[ns1-v6]
Exec[ip addr add 2620:0:861:53::1/128 label lo:anycast dev lo]
Augeas[lo_2620:0:860:53::1/128]
Exec[ip addr add 2a02:ec80:53::1/128 label lo:anycast dev lo]
Interface::Ip[lo-vip-ns2.wikimedia.org-ipv6]
Interface::Ip[lo-vip-ns0.wikimedia.org-ipv6]
Augeas[lo_2a02:ec80:53::1/128]
Exec[ip addr add 2620:0:860:53::1/128 dev lo]
Augeas[lo_2620:0:861:53::1/128]

With @taavi's change in https://gerrit.wikimedia.org/r/q/I5f05c2e4f90dfa8517e7f658216891f46a6a6964, I think we can move ahead with this. If we take eqiad and ns0, we set up ns0 and ns2 (anycast) with bird, therefore they get the loopbacks added with the right labels:

Exec[ip addr add 2620:0:861:53::1/128 label lo:anycast dev lo]

and

Exec[ip addr add 2a02:ec80:53::1/128 label lo:anycast dev lo]

But we also need to add ns1, and that is taken care by the puppetization for the authdns servers itself, 2620:0:860:53::1/128, and this block in modules/profile/manifests/dns/auth/config.pp:

# Create the loopback IPs used for public service (defined here since we
# also create the matching listener config here)
# Skip loopbacks if bird sets up the loopbacks in a given site.
$authdns_addrs.each |$alabel,$adata| {
    unless $adata['skip_loopback'] or $adata['skip_loopback_site'] == $::site {
        interface::ip { $alabel:
            address   => $adata['address'],
            interface => 'lo',
        }
    }
}

Mentioned in SAL (#wikimedia-operations) [2026-02-03T15:32:06Z] <sukhe> sudo cumin "A:dnsbox" "disable-puppet 'merging CR 1230351'": T81605

Change #1230351 merged by Ssingh:

[operations/puppet@production] dnsbox: advertise ns[0-2] IPv6

https://gerrit.wikimedia.org/r/1230351

Mentioned in SAL (#wikimedia-operations) [2026-02-03T17:46:25Z] <sukhe> sudo cumin -b1 -s120 "A:dnsbox and not P{dns1004* or dns7001*}" "run-puppet-agent --enable 'merging CR 1230351'": T81605

Change #1236354 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] wikimedia.org: add IPv6 AAAA glue record for ns1

https://gerrit.wikimedia.org/r/1236354

RIPEstat looks good in terms of visibility of the new ns2 Anycast prefix:

Change #1236798 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] wikimedia.org: add IPv6 AAAA record for ns0

https://gerrit.wikimedia.org/r/1236798

Change #1236803 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] wikimedia.org: add IPv6 AAAA record for ns2

https://gerrit.wikimedia.org/r/1236803

Change #1236354 merged by Ssingh:

[operations/dns@master] wikimedia.org: add IPv6 AAAA record for ns1

https://gerrit.wikimedia.org/r/1236354

ns1 v6 went live today. We will wait for a bit more data to come in but we can see an uptick in connections to the v6 already.

@BBlack suggested that instead of doing unicast for ns[01] like we are doing for v4, we should just have done anycast instead for the v6 from the get go, since the same resource constraints around procuring /24s don't apply to the v6 addresses. That seems fair and something we should have considered in hindsight. That being said, ns1 is already live so we can't change that right now so what's done is done :]

We will need to reassign ns0's v6 so that we can anycast it (from the relevant /48 range in eqiad) and that assignment needs to happen before we can publish the ns0 v6 glue records. ns2 is already an anycast v6, so nothing is required there. Then perhaps once ns0 and ns1 are both up with the v6 anycast, we can start announcing the ns1-v6-anycast, update the registrar and then take down the current, live IP (ns1-v6). Are there any concerns with the plan?

Change #1238012 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: dnsbox: announce ns0 v6 IP as anycast

https://gerrit.wikimedia.org/r/1238012

There is unfortunately one more problem here. Our Puppetization for bird means that we can't set up v6 addresses without v4 ones, in the same configuration. This stems from the simple fact that when we were doing v4 configs, we assumed that those would be the only ones, and support for v6 was added later. Which means we can't do this: announce a v4 unicast IP in eqiad but also have a v6 anycast everywhere else. The reason for this, assuming ns0:

profile::bird::advertise_vips:
  ns0.wikimedia.org:
    address: 208.80.154.238 # ns0 IP, unicast
    check_cmd: '/usr/local/bin/check_authdns_ns0_state /usr/lib/nagios/plugins/check_dns_query -H 208.80.154.238 -a -l -d www.wikipedia.org -t 1'
    address_ipv6: '2620:0:861:53::1' # ns0 v6 IP, unicast
    check_cmd_ipv6: '/usr/local/bin/check_authdns_ns0_state /usr/lib/nagios/plugins/check_dns_query -H 2620:0:861:53::1 -a -l -d www.wikipedia.org -t 1'
    ensure: present
    service_type: authdns-ns0

If we were to make this an anycast address, we would need to address_ipv6 to hieradata/role/common/dnsbox.yaml, but that can't happen because in that hiera config like,

ns0-v6.wikimedia.org:
  address_ipv6: '2a02:ec80:51::1' # ns0 v6 IP, anycast
  check_cmd_ipv6: '/usr/local/bin/check_authdns_ns2_state /usr/lib/nagios/plugins/check_dns_query -H 2a02:ec80:51::1 -a -l -d www.wikipedia.org -t 1'
  ensure: present
  service_type: authdns-ns0

we are not adding a v4, so PCC fails, because:

$advertise_vips.each |$vip_fqdn, $vip_params| {
  interface::ip { "lo-vip-${vip_fqdn}":
    ensure    => $vip_params['ensure'],
    address   => $vip_params['address'],
    interface => 'lo',
    options   => 'label lo:anycast',
    before    => Service['bird'],
  }

So we will need to make v4 optional so that a v6 can exist without requiring a v4.

Change #1238015 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] DNS IPv6 anycast: change router config to support new ranges

https://gerrit.wikimedia.org/r/1238015

Change #1236803 abandoned by Cathal Mooney:

[operations/dns@master] wikimedia.org: add IPv6 AAAA record for ns2

Reason:

git is confusing

https://gerrit.wikimedia.org/r/1236803

Change #1238019 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] wikimedia.org: add IPv6 AAAA record for ns2

https://gerrit.wikimedia.org/r/1238019

Change #1238020 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Change ns1.wikimedia.org IPv6 address to anycast

https://gerrit.wikimedia.org/r/1238020

Change #1238022 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] anycast prefixes: remove old ns1.wikimedia.org IPv6 address

https://gerrit.wikimedia.org/r/1238022

Yeah that seems fine to me.

I've allocated some new /48s from our RIPE /29 range not for these, and created the required ROUTE objects and ROAs.

I think roughly it'd be something like this:

• Make changes in Bird setup to announce just an IPv6 address (as described above)
• Merge homer patch to accept the new anycast IPs from authdns, and announce to the internet
• Create/merge puppet patch to configure the new IPs on all authdns boxes, and announce with Bird
• Verify routes are received, /48s are created and announced upstream, accepted, and that manual DNS requests to the new IPs work
• Add the AAAA records for ns0 and ns2 in our dns / with registrar (see here)
  • I'm not sure if it makes any sense to stagger them?
• Wait to validate they are working as expected
• Change the ns1 AAAA record to point to its new anycast IP (and with registrar)
• Remove the config to announce the old ns1 unicast IP on authdns
• Remove the prefix-list entry on the routers for old ns1 IP

An alternative for better reliability would be to keep at least 1 NS as unicast and served from Liberica. That would save us from a Bird change/bug taking all DNS down.
We grew our dependency on Bird, and with that its complexity (and tooling around it) increased as well. DNS I think is something that should deserve some extra redundancy, and the BGP deamon stack might be one of the lowest hanging fruits to diversify (vs. PowerDNS or gDNSd) especially as Bird is being tinkered on by multiple teams.

In T81605#11600407, @ayounsi wrote:

An alternative for better reliability would be to keep at least 1 NS as unicast and served from Liberica. That would save us from a Bird change/bug taking all DNS down.
We grew our dependency on Bird, and with that its complexity (and tooling around it) increased as well. DNS I think is something that should deserve some extra redundancy, and the BGP deamon stack might be one of the lowest hanging fruits to diversify (vs. PowerDNS or gDNSd) especially as Bird is being tinkered on by multiple teams.

Yeah that's a fair point and something that has been on my mind as well. We have put all eggs in the bird basket and that does worry me. We can somewhat own the bird module and therefore the changes that are merged but that doesn't take away issues with a configuration failure or a bird bug itself.

So yeah, thanks for bringing it up. I will add to our planning and I will discuss with Brandon and Valentin.

Change #1238367 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] nsX.wikimedia.org: publish IPv6 AAAA records pointing to anycast

https://gerrit.wikimedia.org/r/1238367

Change #1238020 abandoned by Cathal Mooney:

[operations/dns@master] wikimedia.org: change IPv6 AAAA records for ns1 to anycast

Reason:

new plan

https://gerrit.wikimedia.org/r/1238020

Change #1238019 abandoned by Cathal Mooney:

[operations/dns@master] wikimedia.org: add IPv6 AAAA record for ns2

Reason:

new plan

https://gerrit.wikimedia.org/r/1238019

Change #1236798 abandoned by Cathal Mooney:

[operations/dns@master] wikimedia.org: add IPv6 AAAA record for ns0

Reason:

new plan

https://gerrit.wikimedia.org/r/1236798

Change #1238375 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] wikimedia NS AAAA: point ns0 and ns1 at dedicated IPs

https://gerrit.wikimedia.org/r/1238375

In T81605#11600407, @ayounsi wrote:

An alternative for better reliability would be to keep at least 1 NS as unicast

There is a discussion to be had on whether to make all the IPv6 Anycast or not as we begin publishing the records. I can see both sides but I'm happy with the "all anycast" idea if that is how traffic want to go.

What might work well longer term is a sort of "partial anycast". Keep the DNS servers the same everywhere - announcing all the /128 IPs to switches - but at certain sites adjust our Homer BGP config so we are not announcing all the /48s from every POP. Which means from a given source network the different NS IPs should route to different POPs - giving us diversity - while not going back to / using site-specific unicast ranges. That setup would also be easy to adjust in routing terms if we need to adapt to a specific situation.

and served from Liberica.

Yeah diversity in terms of how we have it set up is worth considering. I think we should probably separate it from this task however.

Thanks for the feedback folks. There is certainly an intent from Traffic to anycast all v4 and v6 nameserver glues. We have two challenges which we have discussed:

1. For v4 (ns0/1), we will need to get two additional /24 blocks, which we currently don't have, at least not allocated for this purpose. That is something I will start the conversation about with Willy and others, but it won't happen immediately.
2. For v6, the challenge is that our current automation is strongly built around the assumption of v4 and v6 being coupled, which means a v6 anycast cannot exist without a v4 anycast in the same site. We can't fix the v4 bit for the reason above, so the fix is to correct the automation. I plan to work on that ASAP and get that out, incrementally, since this is somewhat of a big change given how many hosts depend on bird.

Given all of this and that ns1 v6 traffic has really not affected anything other than ns1 v4 traffic, I am of the opinion that we should fix the automation this week and push out the v6 anycast when that's ready, perhaps by next week. For me, that is a better approach instead of doing any wrangling right now, dealing with Markmonitor for repeated updates, and we have to fix this anyway so we might as well do it now.

@BBlack and I had a long discussion about this (longer than the usual ones!) and some of it carried over to T366193 where we discussed getting two new /24s for the v4 anycasts.

In the meantime though, we have decided to proceed with v6 unicasts in eqiad/codfw, to match the v4 unicasts there. The reason is that since we will be advertising the glue record for both versions, it probably makes sense to announce them from both sites rather than spreading them out everywhere else if were to anycast them.

Our long-term plan is to pursue one of the design options for splitting up the nsXes as discussed in T366193 but for now we will be proceeding with this change. @ayounsi for your awareness; we plan to do this Monday. I don't think there is any change required on homer's side?

Change #1241003 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] P:bird::anycast: improve the code for IPv6 support (and automatically detect it)

https://gerrit.wikimedia.org/r/1241003

Change #1242423 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] wikimedia.org: add IPv6 glue records for ns0 and ns2

https://gerrit.wikimedia.org/r/1242423

Change #1238015 abandoned by Ssingh:

[operations/homer/public@master] DNS IPv6 anycast: change router config to support new ranges

Reason:

no anycast for now

https://gerrit.wikimedia.org/r/1238015

Change #1242423 merged by Ssingh:

[operations/dns@master] wikimedia.org: add IPv6 glue records for ns0 and ns2

https://gerrit.wikimedia.org/r/1242423

Mentioned in SAL (#wikimedia-operations) [2026-02-24T15:48:56Z] <sukhe> enable IPv6 glue records for ns[02].wikimedia.org: T81605

Change #1241003 merged by Ssingh:

[operations/puppet@production] P:bird::anycast: automatically detect IPv6 support

https://gerrit.wikimedia.org/r/1241003