For posterity:

In T348041#9225405, @ayounsi wrote:

Oops, I missed some of the comments.

• I'm in favor of ditching the statics

In T348041#9225321, @cmooney wrote:

In T348041#9222035, @ssingh wrote:

We can and probably should have a backup static routes for each of ns[01] but it can be to a single host instead of all three.

This is obviously a matter of risk-assessment, and confidence in the various systems at work. But personally I would argue in favour of ditching them completely.

We have 3 listed name-server IPs. Once is anycast from multiple hosts across our pops, the others will be anycasted from multiple hosts at our primary sites. The number of elements that need to fail for user DNS querys to go unanswered is therefore quite high, and I'd come down on the side of them not being needed.

At the end of the day we're talking about a few static routes, however. It's not that onerous to keep supporting it if that is the decision, so I'm happy with whatever the consensus is.

In T348041#9224990, @jbond wrote:

ACAST_PS_ADVERTISE is hardcoded in anycast_healthchecker (the tool we use to monitor services).

in that case agree its to much effort

Thanks everyone for the discussion and feedback above! So it seems like two main points have come up above:

In T347054#9223568, @ayounsi wrote:

I was wondering what to do for all the appliances that have ntp.site.wikimedia.org configured.
To me the best here is to keep ntp.$site.wikimedia.org but have it be a CNAME for ntp.anycast.wmnet instead.

Hi @DennisJJackson: Thanks for the question. We do plan to work on ECH and enable it for our sites and have had some discussions internally. There is no timeline yet as such, for a variety of reasons, the limited browser support being one, though that has clearly changed over the past few weeks. There are some other considerations here as well such as the lack of server-side options for turning it on but we are hoping the DEfO project will provide the much needed support there for HAProxy, which is what we use for TLS termination.

We can and probably should have a backup static routes for each of ns[01] but it can be to a single host instead of all three.

ntp.anycast.wmnet exists and the VIP 10.3.0.2/32 is being announced from all DNS hosts. The next step is to merge https://gerrit.wikimedia.org/r/961812 and attempt a reimage and ensure that it is fine.

In T347054#9203737, @cmooney wrote:

Is there any reason we can't announce the "unicast" IPs in BGP too? I can't really see a good reason that any static routes are needed here.

Thanks for the task! Do we plan to consolidate https://netbox.wikimedia.org/ipam/prefixes/97/ip-addresses/? That is, if we remove the redundant backup static routes, 10.3.0.1/32 can be recdns, 10.3.0.2/32 will be syslog.anycast.wm and then 10.3.0.3/32 will be ntp.anycast.wmnet. I am asking this in the context of T347054 and the allocation of the IP for ntp.anycast.wmnet primarily.

For route 10.3.0.0/30 above, next-hop 208.80.153.77 is actually the old authdns host, so we are clearly not keeping the static routes updated. We should either update all of them or simply remove them so that we no longer have stale records.

In T347054#9203404, @ayounsi wrote:

Thanks, as this VIP won't be critical we can skip the static routes and only allocate 10.3.0.8/32.

The existing "Reserved for XXX (backup static route)" allocations are also slightly wrong in Netbox, I'll fix them.

Looking at 10.3.0.0/24 in Netbox:

Thanks for the feedback everyone! I was waiting so that we can get most of the comments in before replying; responses inline:

Thanks @BCornwall, I think we can close this one as we have done some other reimages in eqsin and not observed this issue.

As per @BCornwall's comment above, it does seem like we have more issue with 9.2.1 that we should look into:

The above patch fixes the issue with CI passing. Once reviewed, we can merge and close this task.

https://meta.wikimedia.org/wiki/Wikimedia_DNS is a detailed introduction of the project, including an FAQ.

Not directly related to bookworm but observed on the dnsdist 1.8.0 upgrade (part of bookworm) that results in a broken latency_bucket metric for the Wikimedia DNS hosts. Reported upstream in https://github.com/PowerDNS/pdns/issues/11239#issuecomment-1707046069.

The hosts were not provisioned in esams but I am fixing that by provisioning them so the durum ones should go away. Thanks for the task update!

I have also tried upgrading the NIC firmware (from 21.40.21 to 22.31.6)

@taavi has rolled this out so this should be resolving shortly. Thanks for filing the task.

Thanks @Michael for filing this task. Restarting zuul helped a bit but only for a short period of time, after which the failed CI issue was back again.

The requested records have been updated. Thanks!

In discussion with @cmooney, we will be revisiting this task again when Traffic does some other authdns-related work, so removing it from the Traffic-Icebox.

In T342159#9025176, @RobH wrote:

Please note parent task 341588 has the range of cp1[090-105] however, cp1090 is already live/in use. Additionally, we have 4 cp hosts from eqsin to use for CP in eqiad (so cp109[1-4], and another 4 coming from ulsfo cp109[5-8).

Also no racking info has been provided for these. Should they be distributed in any particular way for traffics needs/redundancy?

Assigning to @ssingh for followup on racking details and to clarify hostnames (and ensure the 8 old CP hosts from ulsfo/eqsin are still expected to be used.)

@thcipriani and @odimitrijevic/@Milimetric this requires your approval for the deployment and analytics-privatedata groups respectively.

Hi @NHillard-WMF: This will require approval from your manager. Also adding @odimitrijevic and @Milimetric for approval on the Analytics side.

The link you shared is correct and no SSH keys are required. We will wait for @NHillard-WMF to confirm if he requires more access, otherwise this ticket itself is sufficient for SRE.

@Fabfur and I observed the same issue on trying to reimage lvs1016.

pdns-recursor 4.8.4-1+wmf11u1 has been running in production on the following hosts for a while:

Affected hosts:

Traffic has commissioned these boxes. Many thanks to dc-ops!

The hosts have been decomissioned and ready for the hardware part.