In T385404#11356077, @ori wrote:

@akosiaris Do you have any suggestions for getting this task un-stuck?

In T352956#11247042, @ssingh wrote:

Hi @akosiaris: Following up on this after a discussion during Traffic's planning with @Vgutierrez, and on behalf of the team.

We were curious to know when you would be able to take this, with the understanding that things are busy and we don't expect it to happen immediately. From Traffic's end, we have decided to triage this for Q3 for the rollout and not Q2, given that it is short quarter and we are unlikely to roll such a big change before December that will affect the core sites. (Liberica is already running on all PoPs.) So that's at least our position.

Does that seem fine per you and the planning for Serviceops? Do note that as per @Vgutierrez's last comment, we already have a check in place in the migration cookbook so you do not have to worry about that.

Thanks!

In T408704#11326432, @LSobanski wrote:

cc @akosiaris

And this is the yamldiff of the same upgrade

Does "we" include every SRE in the SRE department? Or is that only the "tools-infra" team? Or is it "tools-platform" + "tools-infra" teams?

In T407296#11274680, @Andrew wrote:
"After building a pilot bare-metal toolforge replacement, how hard was it, and does it seem like something that would be easy to maintain, or hard to maintain?"

In T401895#11225201, @MSantos wrote:

@akosiaris per T392491#11167986 and message sent to Wikitech-l earlier Today, this is ready to go.

In T404726#11204802, @dcaro wrote:

In T404726#11202598, @JJMC89 wrote:

In T404726#11202498, @dcaro wrote:

Hi @JJMC89, thanks for your comments.

If you are going to change the CPU/memory defaults or what the CPU and memory values translate to in k8s requests/limits, please communicate it in substantially in advance of deployment.

Can you elaborate on this? Specifically on how does it affect your workloads? (there's some changes that are more time sensitive than other and might be needed without much advance notice, for example the cpu requests/limit defaults that's already hitting the cluster allocation space)

I do have some jobs that are time sensitive, but my concern is maintainers not being available to adjust job resources before you deploy this.

In T404726#11202421, @dcaro wrote:

An issue here is that a lot of our workloads are cronjobs, so cronjobs being unable to trigger means most of the users don't get their workloads running, so I think that not being able to schedule new workload is a critical enough situation to grant a page (at least in the current situation, where we can expand the cluster at will to palliate it in the short term). We can discuss in the team meeting also.

In this scenario, are we envisioning a situation where cpu requests > allocatable space with or without the cronjobs? If with, then I argue that we will be seeing is a delay in starting workloads, not an inability to have the workloads run. Which again, isn't paging worthy (but it definitely needs an an alert, probably critical). If without, I have to ask what are the scenarios that would trigger this. Lost so many nodes that we are out of capacity? Some how we scheduled so many non transient tools that

I think there's some part of the phrase missing :)

In T404726#11200892, @dcaro wrote:

Thanks @akosiaris, this is very helpful :)

So I propose then for limits/requests to do:

• If the user specifies --cpu/--memory, use that as both request and limit (as the expectation is to use that, no more, no less).
• If not, then:
  • cpu/request: use a default that is the average of the cluster (currently ~7% currently, that'd be 70m, we can round to 100m for now)
  • cpu/limit: use a big number, smaller than a node, we can do something like 4000m (4 cores, as the workers have 8 cores right now)
  • mem/request: use a small, but not too small number, currently it's 256Mi (limit is 512Mi, we use half of it), though we are using ~60% of the actual memory of the cluster, and the requests are around 80% full, we could make it smaller but I think it's kinda ok already
  • mem/limit: this one is trickier, as we know that toolforge workloads are usually very spiky here, with sudden memory usage and then quiet for a long period. Setting this to the same as the request means that we will never overcommit memory-wise any node, but also that most of the time the memory will not be used and any user having bursty workloads will have to specify the maximum memory used manually. Currently we set it to 512Mi (double the request, well, the request to half the limit).

Then for the alerts:

If the cpu requests is bigger than the allocatable space, then users will not be able to get their pods scheduled anywhere, so they will stop running -> page?

Most workloads will continue churning along. It's new ones that won't be able to be scheduled. I 'd argue that is not a page. It's a degraded experience, sure, but a critical alert is good enough, IMHO.

An issue here is that a lot of our workloads are cronjobs, so cronjobs being unable to trigger means most of the users don't get their workloads running, so I think that not being able to schedule new workload is a critical enough situation to grant a page (at least in the current situation, where we can expand the cluster at will to palliate it in the short term). We can discuss in the team meeting also.

I 've already replied in T394982#11201112, but I find it improbable that SRE will be implementing such a behavior to accommodate for the change in node.js fetch() API. The HTTP Host header is pretty important across the infrastructure. Rewriting other HTTP headers to it might make debugging and reasoning more difficult than needed.

In T394982#11142262, @santhosh wrote:

Node 22 stabilizes the fetch API. It is now feature compatible with browsers fetch API. This is generally good, but it also adds more restrictions to what a valid http request can be.  The header field we are setting to pass the wikipedia domain to that wiki proxy is HOST (see the configuration). This is problematic because HOST is a forbidden header.

1. https://developer.mozilla.org/en-US/docs/Glossary/Forbidden_request_header
2. https://fetch.spec.whatwg.org/#forbidden-request-header

So, the nodes fetch API wont accept HOST header. The wiki proxy will recieve the request without the HOST header and will end up 404 response.

In T404742#11197487, @CDanis wrote:

In T404742#11194509, @akosiaris wrote:

I still have the same concerns as voiced in T359067#9602091, but I also have to be pragmatic. I don't see us solving the bigger registry problems in the next 6 months

Furthermore, Dragonfly helps mitigate some of the concerns. Some more operational ones I have are:

• Experience has shown that VMs with a lot of memory are more difficult to migrate around in Ganeti, some to the point of stalling. However, I am thinking it shouldn't be any stalling (we 've seen that with VMs with high continuous memory churn and this use case isn't one of these), just taking longer.

Yeah, historically true. I'm hopeful (perhaps naively) that it's better nowadays:

• 10Gbit is almost everywhere, as Moritz says
• most Ganeti metal is 128GB RAM, so the schedulability of these large VMs is also easier
• in addition to the lack of RAM churn given their use case, these VMs are also totally fine to shut down one at a time and cold-migrate

Thanks for this writeup! Couple of inline replies

I still have the same concerns as voiced in T359067#9602091, but I also have to be pragmatic. I don't see us solving the bigger registry problems in the next 6 months

Setting to stalled, while we figure out the exact details of this one.

In T390438#10804169, @Magog_the_Ogre wrote:

No, the bot just hits the categorymembers API.

We already reopened this once. Maybe we wait a bit to monitor before closing?

I 've gone ahead and shaped up some older notes I had in wikitech and posted a guide for capacity planning in https://wikitech.wikimedia.org/wiki/Kubernetes/Capacity_Planning_of_a_Service

Mentioning this here as well as in T403094: Request to increase function-orchestrator memory to 10GiB. I 've gone through the logstash entries and the related kernel logs and events and there are only 2 instances where the kernel OOMKiller showed up and killed a container and even in that case it was the mcrouter container. I think the issues in this task are a combination of

The orchestrator is crashing very frequently now due to OOM

In T399348#11010993, @bd808 wrote:

In T399348#11009805, @Dzahn wrote:

We could possibly discuss with releng if Digital Ocean runners should still be default or maybe the other way around, default to wmcs and people can opt-in to Digital Ocean runners if they want. (Given that we had a few reports where the suggested solution ended up being to switch to our own infra.)

The WMCS runners have less functionality than the DO runners (T397888, T396924), so we would probably want to at least make these differences well documented and warn everyone of CI failures to be on the lookout for if the default is changed.

In T401269#11086333, @cmassaro wrote:

In T352956#11083016, @Vgutierrez wrote:

In T352956#11016142, @akosiaris wrote:

Yup, scheduling it for the weeks of either August 11th or August 18th.

gentle ping, do you need something from my side?

Thanks @claime. Stupid typo on my side.

What does backend refer to here?

In T390251#11084257, @thcipriani wrote:

@akosiaris looks like you had a lot of changes for making a new registry, is that registry ready? Or is it still in the testing phase?

Re-reading my first response, I realize I might have been a bit unclear. Indeed my focus was to respond to number 2, namely should it contain the shell username of whoever made the wiki? , not to question sending the email in the first place. I agree that this should still be sent. It's a notification as you say, not an auditing mechanism.

Patches deployed, you should be good to retry @cmassaro

In T401833#11082991, @Jdforrester-WMF wrote:

In T401833#11082897, @akosiaris wrote:

Those are warning (note the W prefix). They wouldn't stop the deployment from happening.

The actual reason is this https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.08.13?id=8Q7Uo5gBgiE0yhV9mhEm

Pasting for convenience

(combined from similar events): Error creating: pods "function-orchestrator-main-orchestrator-df5fdb7c9-dmrg5" is forbidden: exceeded quota: quota-compute-resources, requested: limits.memory=4172Mi, used: limits.memory=7220Mi, limited: limits.memory=10Gi

Simply put, the namespace in staging isn't provisioned for pods that are this large. Resources in staging are scarce. We can probably bump the quotas up a bit, but probably just enough to allow this to run.

Aha, we can pull the memory for staging back down.

Those are warning (note the W prefix). They wouldn't stop the deployment from happening.

BTW, on the technical side, mw-script does indeed keep the username in the labels of the job and the pod, e.g.

This functionality was added 10 years ago in https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikimediaMaintenance/+/a17c2ef30e0e85ced460f304cf481cdb7d924486%5E%21

In T352956#11015954, @Vgutierrez wrote:

@akosiaris I think we could start considering enabling inbound IPIP traffic on the staging environment, deploying IPIP interfaces (assuming you'll be using the regular kernel networking stack and not some eBPF "magic") shouldn't affect the ability to handle non-encapsulated traffic.

As soon as IPIP encapsulated traffic is handled we can validate that it's working as expected without impacting the traffic coming from load balancers, we used sre.loadbalancer.migrate-service-ipip cookbook to perform this validation for T373020: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/loadbalancer/migrate-service-ipip.py#182 and we could use a similar one here

In T399681#11013197, @A_smart_kitten wrote:

Hmm… what's interesting about this task & the other two most-recent MediaWiki-Special-pages cron-job-failure tasks (T396454, T396977) is that all three of these failures have been due to (what seem like) database connection errors.

This was probably due to https://sal.toolforge.org/log/huohd5cB8tZ8Ohr03499

There you go.

We don't have strict requirements around the intra DC availability zones.

Thanks for tagging me in this one. This is more Infrastructure-Foundations territory these days, so I am adding the relevant people as well for their information.

In T361768#10995844, @tchin wrote:

In T361768#10995389, @akosiaris wrote:

In T361768#10459087, @Ahoelzl wrote:

This includes upgrade to Nodejs 20.

Hi! Has this happened? Looking at the images currently deployed per deployments-charts repo

$ podman run --rm -it --entrypoint /bin/sh docker-registry.wikimedia.org/repos/data-engineering/eventgate-wikimedia:v1.11.0 -c "nodejs -v"
v20.5.1

and

podman run --rm -it --entrypoint /bin/sh docker-registry.wikimedia.org/repos/data-engineering/eventgate-wikimedia:v1.14.0 -c "nodejs -v"
v20.5.1

says yes, but I guess it doesn't hurt to double check that we are all on the same page.

Yes, this ended up happening separately with T383814

In T361768#10459087, @Ahoelzl wrote:

This includes upgrade to Nodejs 20.

No new reports, I 'll resolve, feel free to reopen.

All kubernetes clusters are now configured to use MTU 1460. This will take some time (weeks) to fully propagate, as this requires a pod restart. Deployments, node maintenance, evictions and other events that end up restarting or rescheduling pods will trigger it. In a few weeks we should be in a position to look at the few left hanging fruits and manually restart those.

wikikube workers repooled.

In T398433#10974433, @ayounsi wrote:

Sweet, what about 12:00UTC on Monday 7th ?

The 2 patches have been merged and will ride out today's deployments. Hopefully we 'll be able to successfully resolve this task next week.

I 'll file a patch though to increase the maximum bucket.

In T380544#10952106, @LucasWerkmeister wrote:

Okay, would there be a problem with running more refreshLinks jobs across all wikis? 😇

I have https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1164269/2 and https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1164270/2 lined up for deployment on Monday, they should resolve what is witnessed by deployers.

Response from the redfish API, using my gerrit change above to print out the response

In T380544#10929363, @LucasWerkmeister wrote:

I guess the most likely explanation is that the processing of the updates for MariaDB and OpenSearch happens separately, and the MariaDB updates to the templatelinks table are progressing rather more slowly. Between T380544#10346771 (79.1M) and T380544#10924765 (51.5M), almost seven months passed; based on this, we can extrapolate that the remaining 51.5M rows would take another year or so to process. (Or, if we assume the CirrusSearch updates are complete and the 16.9M pages there still have genuine uses of Template:SDC statement has value, then there are 34.6M rows left which would take between 8 and 9 months at the current rate.) That’s not as bad as the “ten years” estimate, and certainly not bad enough for High priority, but I still feel like a somewhat higher rate of these jobs wouldn’t hurt.

Our first arm64 server just got racked. We 'll need to figure out how to incorporate it in our tooling, see T397653, but we are finally moving.

In T393015#10939425, @Jhancock.wm wrote:

This server isn't getting a clean provisioning run

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
  raw_ret = runner.run()
            ^^^^^^^^^^^^
File "/srv/deployment/spicerack/cookbooks/sre/hosts/provision.py", line 294, in run
  self._config_host()
File "/srv/deployment/spicerack/cookbooks/sre/hosts/provision.py", line 433, in _config_host
  bios_attributes = self._get_bios_settings()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/deployment/spicerack/cookbooks/sre/hosts/provision.py", line 369, in _get_bios_settings
  return bios_settings["Attributes"]
         ~~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'Attributes'

Copy pasting from the commit message of the change just merged and deployed

Judging from the lack of comments in the last 2 weeks and repeated tests by yours truly, there is a chance the approach worked on the mwdebug hosts (which are slated for removal anyway), so moving forward this week with the mwdebug on k8s as well.

In T380544#10924765, @LucasWerkmeister wrote:

Overall, I’m inclined to say that by now it’s safe to just close this task and assume the job queue is doing its job reasonably well.

In T393557#10918072, @ihurbain wrote:

Note: I actually do not know how these are generated, so it's plausible that it's expected as long as the end point still exists even if it's returning 403 - but https://en.wikipedia.org/api/rest_v1/#/Page%20content is still documenting the existence of the endpoint.

Thanks, this helped. After creating manually the registry-restricted (docker-registry will return 503 if it doesn't exist) bucket with s3cmd, and some temporary hacks to prove this works, I managed to partially push an image.

In T393557#10904336, @MSantos wrote:

@akosiaris this is ready to for traffic blocking. Please let us know if that make sense or if you need any other information.

I 've gone ahead and switch all of aux-k8s to MTU 1460. This time around, I went for a more hands off approach, namely:

In T352956#10900403, @cmooney wrote:

@akosiaris a quick question about this:

meaning that ICMP traffic to e.g. coredns gets dropped

In terms of pmtud that means that if coredns sends large UDP packets - which get dropped elsewhere - it won't get the ICMP "packet too big" messages back. But that is not really a worry. The CoreDNS PODs have a lower MTU than pretty much everything on the network, they are not going to send packets that are too large for anything else.

Tried deploying this, see https://sal.toolforge.org/log/drexRZcBvg159pQr5-r8 and logs have the following

Indeed, I commented in the wrong task. My mistake, thanks.

I 've run a battery of tests in the previous days against mwdebug2001 and mwdebug2002, having 2 different configurations regarding retry_on policy. mwdebug2001 had connect-failure and mwdebug2002 had 5xx. The latter includes the former for what is worth. The tests were just invocations of the command in T380958#10887212. mwdebug2001 had the occasional fail transaction in a number of those tests, mwdebug2002 has consistently not return an error. Thus, https://gerrit.wikimedia.org/r/1155117 which I just merged. Let it it soak for the week. Should this work OK, it needs to be backported to the configuration of mwdebug on k8s as well.

Logs have the following

E20250606 14:46:30.010773     1 Server-inl.h:593] mcrouter error (router name '11213', flavor 'unknown', service 'mcrouter'): Failed to configure, initial error 'Failed to reconfigure: Unknown RouteHandle: KeyModifyRoute line: 81', from backup 'Failed to reconfigure: Unknown RouteHandle: KeyModifyRoute line: 81'

Thanks for this! I 've also landed a round of updates today in https://wikitech.wikimedia.org/w/index.php?title=SLO/Template_instructions/Dashboards_and_alerts&diff=prev&oldid=2309464.

Switch to stalled while waiting for the release.

Close to 100k requests later to Main_Page of spcom.wikimedia.org, 1 error. This is below very close 99.999% btw. Siege rounds up to 100%, but actual availability is 99,998% which is very very very (did I say very enough?) good.

I 'll be hammering mwdebug2001 with siege for the next 30 minutes in an effort to reproduce this. Command is run from deploy1003 and it's

In T380958#10885815, @cjming wrote:

encountered this deploying today - https://spiderpig.wikimedia.org/jobs/154

not sure if it was a blip in the matrix or related but retrying testserver checks seemed to resolve it

This is surfacing once every couple of days or so, at least per Logstash, which counts 23 instances in the last 2 months. I was looking at one that @Marostegui met today.

Thanks @ecarg. I 'll defer to @mszabo for reviewing. He is definitely better equipped currently than I to review.

In the interesting of stopping the non actioned upon alerts, I 've gone ahead and excluded linkrecommendation from this alert. Patch is https://gerrit.wikimedia.org/r/c/operations/alerts/+/1150726. I 've also it from serviceops to serviceops-radar. Feel free to undo when the time comes that someone works on it.

In T390517#10692685, @akosiaris wrote:

Per https://w.wiki/DeQh, we are well within our estimations about the day amount of requests still reaching this service. It's in the order of ~1900 and we had estimated ~2k.

In T391333#10853430, @elukey wrote:

In T391333#10852614, @akosiaris wrote:

I 've spent a good deal of time today to do what I assumed to be easy, that is perform the above in our ingressgateways. I have failed up to now. I 'll need to revisit this with a fresh mind, cause right now I even have doubts the ISTIO_METAJSON_STATS trick works.

@akosiaris I added some thoughts to T392886, lemme know what you found. My understanding is that there is a new annotation that we could use in recent versions of Istio (1.24 has it, but it cannot easily be backportable to our version afaics), I am very open to new roads to check!

If you want to do some testing, I could set you up with a test account on apus.

1 single bucket, at least at the beginning. Reading https://distribution.github.io/distribution/about/configuration/, I don't think the software can use more than 1 bucket anyway. We could in the future, assuming the PoC ends up being successful, to discuss splitting strategies to have >1 bucket. But even in that case, since that requires running an extra instance of the software, per my current understanding at least, we are limited to a small number, probably in the single digits.

I 've spent a good deal of time today to do what I assumed to be easy, that is perform the above in our ingressgateways. I have failed up to now. I 'll need to revisit this with a fresh mind, cause right now I even have doubts the ISTIO_METAJSON_STATS trick works.

After having to deal a bit with a staging-eqiad calico upgrade yesterday, I did find 1 thing that will break. This is a bit complex:

I did run the simple one

staging-eqiad with an MTU of 1460 as well.

In T352956#10839001, @akosiaris wrote:

Long-term

We probably want to minimize our diff from upstream manifests in order to allow easier upgrades in the future. We can gradually move away from managing CNI in puppet now that upstream has support for this, making long term our lives easier.

Unfortunately the 2 patches above didn't work. For ml-staging-codfw, just because it's still, via virtual of helmfile.d/admin_ng/values/common.yaml, locked to 0.2.10. However, it did not work either for staging-codfw because while the upstream manifests do indeed have support, that support is implemented via having the CNI config being managed by calico, a support that we did not import in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1112058, sticking with CALICO_MANAGE_CNI: false. We do manage this via puppet. It should be noted that our puppet implementation does allow to differentiate per cluster, same as the chart approach. There is no really functional difference between the 2 ways.