Update: we are still waiting to get clarity on what is happening here. We are pursuing this and I will update this when things change.

In T427465#11994850, @Ladsgroup wrote:

Contrary to what it looks like, I have no clue what I'm doing. We probably need a service in hieradata/common/service.yaml too but not 100% sure

In T428060#11993627, @ayounsi wrote:

@ssingh For the DNS servers, the ones peering with the core routers will have a higher priority (as-path) than the ones peering with the ToR switches. So if one can handle all the load then we're good as we won't have any redundancy issue.
We can use as-path prepending to fine tune it, but if we can avoid it it would be better.

Plan could be:

• Move dns2004 - internet load will be shared on 2005/2006 (making 2004 roughly a hot standby, while still receiving 10.3.0.1 queries from its pod)
• Test that 2004 works well
• Move dns2005 - all the internet load will go to 2006
• Shortly after, move dns2006 - all the internet load will be balanced between the 3 hosts again

dns[2004-2006].wikimedia.org - Need special care to not cause traffic imbalance @ssingh

In T414411#11986915, @BCornwall wrote:

We discussed this and the general consensus seemed to be to just decomm the server and wait for the refresh which is happening shortly anyway. @ssingh Is that accurate, and if so, ready for me to open a task/decom it?

In T428093#11982863, @taavi wrote:

In T428093#11982860, @ssingh wrote:

So payments is a CNAME and hence we can't add a CAA record for it.

Can we not add a CAA record to the CNAME targets (payments-eqiad/codfw)?

CAA works at the subdomain level so we can set the records for payments.wikimedia.org to allow them their issuance while removing the unnecessary records for the rest of the stack.

In T428052#11982183, @bd808 wrote:

The Beta Cluster cache nodes are Debian Bullseye running HAProxy version 2.8.18-1~bpo11+1 2025/12/26. It looks like the prod CDN edge is using Debian Trixie and HAProxy 3.2.15-1~bpo13+1.

See also: T401839: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie

In T117618#11974306, @sbassett wrote:

Revisiting this over the past month, it looks like we're receiving, on average, ~ 2500 report-only reports each day for the current upload.wikimedia.org CSP config:

@ssingh - I think we should remove the restrictions/filters and just serve this as a report-only policy across all of upload.wikimedia.org, for all media files, for a few days. And then ultimately set it as an enforcing CSP. At this point I'm not seeing any compelling reason not to aggressively move forward with this either in the comments here or CSP report-only log data.

Update: We are looking into this -- on how to update the list of Bing IPs -- and will update the thread.

In T425850#11968618, @AlexisJazz wrote:

The request was simple. Look in the server logs.

In T426912#11964942, @BCornwall wrote:

@ssingh @BBlack Okay with me switching write-back to write-through slowly through the codfw cluster or shall we leave this as-is until the refresh?

In T426968#11965021, @BCornwall wrote:

@ssingh What made you suspect mem errors? I see from the previous boot that OOM kept getting invoked on purged but I suspect that from some non-hardware issue.

{F84961765}

I meant we set profile::netconsole::client::ensure: absent in hieradata/role/common/cache/upload.yaml so it should not be enabled there.

Yeah it's a good question and predates me so I don't have good answers. But, it doesn't seem that it is enabled for upload? The upload role does say include profile::netconsole::client but we also set profile::netconsole::client::ensure: absent and it seems like nothing is being actually set on upload unless I am mistaken.

In T427491#11964208, @daniel wrote:

In T427491#11963857, @ssingh wrote:

Per https://www.mediawiki.org/wiki/Manual:Pywikibot/User-agent, users can set a custom UA easily by setting user_agent_format. Should we perhaps not be encouraging that vs blanket allow-listing the default UA? That then falls into line with our other UA policy for library defaults -- each bot should set a custom UA.

The default UA is customized automatically and contains contact information based on the user's login info. Why should the bot'S owner have to override the user-agent string if they are already supplying their user name? All we want to know is who is running the bot.

I'm not suggesting to whitelist pywikibot, I'm suggesting to support the format in which pywikibot supplies user information in the user-agent in accordance with our policy.

Per https://www.mediawiki.org/wiki/Manual:Pywikibot/User-agent, users can set a custom UA easily by setting user_agent_format. Should we perhaps not be encouraging that vs blanket allow-listing the default UA? That then falls into line with our other UA policy for library defaults -- each bot should set a custom UA.

The host is acting up again and is depooled. It seems like this time it is memory errors but racadm points to nothing. I am going to leave it depooled while we look.

Depool for cp2044 looks good; please ping Traffic if you want us to take care of it.

In T426109#11957496, @BCornwall wrote:

@ssingh The Dell docs mention updating the BIOS:

update BIOS to the latest revision that includes many memory Self-healing capabilities and ongoing enhancements

However, that would put our version ahead of the rest of the fleet. Is that acceptable or should I just keep it as-is?

Once we reboot for T426585, we can consider this resolved as well.

Thanks for the update and the explanation, @cmooney!

This hasn't happened in a while (last incident was 2023) and we have run sre.cdn.roll-reboot many times since then, so boldly resolving.

@cmooney: We plan to move to Liberica in Q1 or Q2 of APP2026. Do you think we should still consider working on this?

This hasn't happened again and it's hard investigating now what caused these two blips. Boldly resolving for this as part of regular task cleanup. If it happens again, we can look into it.

Boldly resolving for the reasons above: the issue was transient because we responded to it, there is no follow up and there is nothing on our side to indicate that this persists.

We never got to this in Q3 or even Q4. Should we plan to do this in Q1 2026?

LVS in core sites will be superseded by Liberica so we are unlikely to spend any time on this. I am taking the liberty to close this as part of regular task cleanup, please re-open if desired.

No one else has observed this issue and it has been almost two years since this was reported, with no follow-up. As the person who reported this, resolving.

It seems like the issue was transient and therefore I am taking the liberty to close this as part of regular task cleanup. Please re-open if desired.

I am curious, which error pages are we talking about?

@Ergur: Hi. Can you confirm if this is a problem for you still or has resolved?

All hosts were reimaged and have been back in production for a while. Resolving.

LVS in core sites will be superseded by Liberica so we are unlikely to spend any time on this.

There has been no follow-up to this in a while (and this is on k8s anyway now?) and this task has been open since 2024. I am taking the liberty to close this as part of regular task cleanup, please re-open if desired.

The only blocker in this task was the cp hosts for OpenSSL. We have already upgraded them to trixie in T401832, so this task can be resolved. A note has been made for the same in the description.

This has been completed in various iterations of the updates to geo-maps.

We have done quite a few reimages of durum since then (and reboots) and this issue was not observed. I am taking the liberty to close this as part of regular task cleanup, please re-open if desired.

In T414411#11945980, @RobH wrote:

Without getting into pricing on this public task the options are:

• spend more money (see T426985) to replace the CPU
  • we have no money left in expendables for this, so it would likely have to kick to July (unless mgmt approves overspend)
• shuffle all the memory to report to the working CPU and use this CP host with only half the CPU cores
  • no clue if this is viable, this is a question for @ssingh
• decommission the host and use as spare parts for rest of eqsin fleet
  • worst case scenario since it means a depreciated fleet quantity in eqsin.

In T414411#11945112, @RobH wrote:

Apologies, this ran super late and I neglected to update the task accordingly.

The mainboard swap was successful but it appears of the two CPUs, one of them has failed. Dell SG is sending over a quote for replacement, as the system is currently remotely accessible with only one of two cpus installed.
I'll update the task with the new quote later today!

In T414411#11914980, @RobH wrote:

Scheduled a new site visit for them to go out this Friday @ 8AM Singapore Time so my Thursday @ 4PM.

1-260037210462

Should now be resolved; @bd808 already cherry-picked but this has been rolled out to operations/puppet as well.

Can someone confirm on how to reproduce this? If I try to go to Bing and do a reverse search with Commons, it seems to work for me.

Without knowing the details of this, I wanted to point out that the drmrs refresh is upcoming in Q1/Q2 of FY2026 and drmrs like all edge sites, is on Liberica. If there is any redesign we want to do around this in drmrs, perhaps this is the time?

Yeah I should have been more careful in resolving this, my bad. @Jhancock.wm: While the DIMM was replaced, we still need to look at the RAID thing.

Hi: Following up to see if this issue still persists; I think perhaps not (see my comment above) since I think it was transient but please let us know.

Hi @BrokenImages1234: Following up to check if this issue still persists for you?

@sbassett: I am having a bit of trouble parsing what exactly is failing, but if it is just the report-to header, note that we set that in Varnish (VCL) for Network Error Logging. That looks like, for example,

Things look good and lvs2012 is happily serving traffic. Marking as resolved, thanks @Jhancock.wm!

Confirmed with @Papaul that this is actually meant for May 20, same time.

In T416562#11914213, @Papaul wrote:

@ssingh i think it will be best to depool the site since this will be my first time doing the draining process I will like to be on the safe side.

In T416562#11914182, @Papaul wrote:

@ssingh and team now that we are done with the switch refresh and everything is stable in ulsfo and after we connect the missing link between cr3 and asw1-23 We will like to schedule a 3 hours downtime for the JUNOS upgrade on the core routers next week May 14th at 9:45am CT 10:45am EST. Please let us know if this time and date works for you.
Thanks.

In T425120#11913059, @Ponor wrote:

@ssingh
Thanks. This has been an issue for some 10 days, but I haven't noticed it recently. I see that FF got a little update, so it's hard to tell what the cause was.
I thought it'd be good to report it in case someone else was having the same issues. I'll close the ticket now.

@VRiley-WMF: As John correctly pointed out, this is booting with UEFI enabled now. Is that expected and the default for all hosts now? If that is the case, we can update the partman recipes but this is an old host, so I am surprised that UEFI is the default. Any insights on that?

Thanks for the report @Ponor. Does this issue still persist? If yes, can you please undertake the steps in https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue and reach out to us at noc@wikimedia.org with the output? You can refer to this task.

In T421421#11909801, @Jclark-ctr wrote:

@ssingh you are booting with UEFI?

the YAML file need to be updated for lvs1017

-partman/standard-efi.cfg
-partman/raid1-2dev-efi.cfg

Hi @BrokenImages1234, thanks for your report. The error report you indicated, 76af2b0, does indeed point to the issue on why you are seeing 429s. Upgrading to a more recent browser version will help fix the problem. Also, you can try enabling third-party cookies and that should help improve the rate-limiting somewhat.

I may be wrong but this was due to a temporary issue we had with upload.wikimedia.org in ulsfo, which matches the time of this report, and also matches the traffic from New Zealand, since that goes to ulsfo as well. I believe that this should now be resolved.

@VRiley-WMF: We may need to check this host; I can't seem to get it to come back up after a reboot (checked twice). Is there something else missing here? Perhaps a provisioning cookbook run or something? (Just guessing!)

In T421421#11909017, @VRiley-WMF wrote:

@ssingh I have almost gotten it all the way through. However, it doesn't seem to take the reimage. It's seemingly is getting stuck at the raid. I tried to log into the other other lvs servers however, I'm unable to. Is there a specific raid this needs to have? Let me know, thanks!

@KOfori is out, deferring to @Kappakayala as the approver in the interim.

@Jhancock.wm: Thanks for the quick turnaround! Host is back and serving traffic, will keep a close watch for a bit before resolving this.

In T425890#11908254, @Jhancock.wm wrote:

i pulled a replacement DIMM and a ssd from our offlined hosts.
@ssingh safe to power down the host?

In T421421#11907442, @VRiley-WMF wrote:

Hey @ssingh Is it okay to make this change today?

Yeah, no, I was wrong and I misunderstood the problem. I misread that the actual image also differs across the CDN but it clearly does not.

In T425216#11887650, @TheDJ wrote:

I believe there is a 24 hourly script that checks cross dc consistency or something

^ The above is not a trixie reimage but a bookworm reimage for the ulsfo work in T424686. Please disregard.

In T408892#11749076, @ayounsi wrote:

As a side note we will need to manually change the IPs of the routed ganeti nodes in rack 23 to the 10.128.1.0/24 subnet. Normal operation would have required a re-image but to not lose the VMs and make the migration faster it's best to re-IP the hosts.

AFAIK. the last thing needed is to convert those BGP policies from Junos to Nokia:

• Anycast4
• Anycast6
• Ganeti4
• Ganeti6
• Management
• PyBal
• Core

In T424549#11862117, @elukey wrote:

@bd808 yep it is, I missed this. I think we can spin up discovery2026 on cloud as well for consistency, I'll try to do it tomorrow!

Slight correction on my end, sorry: this is host is not an active host, so you can install the NIC whenever you want before May 11 as well. We will however pick this up on our end during that week.

In T420604#11855938, @sbassett wrote:

In T420604#11852692, @ssingh wrote:

For 2), we will be removing the VCL altogether I suppose? And you meant "same changes" as in same changes on MW?

Yes, remove the VCL CSP for Wikimedia production entirely. Which I guess is just removing it entirely. Except for any odd exceptions for static assets, etc. that may already exist (I think there might be one for PDFs and/or testwiki). And then Wikimedia production will be (mostly) controlled by MediaWiki's CSP implementation. So these operations really shouldn't affect what user's experience now except for the report-only header going away. But in the future we plan to define more CSP directives and further tighten the current CSP allow-list - these steps just get us further down that path.

@VRiley-WMF: We are planning to do this the week of May 11. Does that work for you?

Hi @RobH. Any update on this from Dell's end?

It seems like @SKaram-WMF's Kerberos credentials were never created initially:

In T420604#11852000, @sbassett wrote:

So I think the next steps here are to:

1. Monitor beta and check beta-logstash over the next few days to ensure stability
2. Make these same changes within Wikimedia production

Discussed with @Papaul a bit -- we will depool the site for all three days, just to be on the safe side and since it's ulsfo, one extra day does not really change things.

In T420604#11846467, @sbassett wrote:

In T420604#11845454, @ssingh wrote:

We can selectively remove the filter in Beta if desired, yes. Right now there isn't a proper way of doing this other than using the hacky etcd_filters variable. I will prepare a patch.

Ok, sounds great. Happy to help test/confirm once that's deployed.