We know from T405224 that we can recover the entire cluster from the K3s data volumes. Additionally, the folks over at Cloud have told us in the past that it's not possible to automate/schedule the creation of OpenStack snapshots natively.

As usual, we will announce a maintenance window for the migration.

Operation times in production have significantly improved and gone back to the performance levels we had by the end of November 2025.

Similarly to what happened with secrets, recreated envs have been leaving behind a significant number of replica sets behind:

kubectl -n cat-env get rs | grep 750c4d946d
wiki-750c4d946d-3895-mediawiki-66bccd4d77   0         0         0       15d
wiki-750c4d946d-3895-mediawiki-8669b898df   0         0         0       15d
wiki-750c4d946d-3895-mediawiki-668466c6b8   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-c556c45c5    0         0         0       14d
wiki-750c4d946d-3895-mediawiki-76c5fb6d87   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-6968d99c4d   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-6b5b86ff6f   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-68cd949987   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-75749d974b   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-5d46bb4dc5   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-68c5994db9   1         1         1       11d

Some of the helm commands had started slowing to a crawl; as it turns out we had 912 secrets in the cat-env namespace. A majority of those were originating in helm history revisions. A lot of those revisions could be safely removed and after doing that helm commands became noticeably more responsive again. Plus env creation times have gone down again to the levels we had a few months ago:

In T414659#11632424, @A_smart_kitten wrote:

In T414659#11631948, @jnuche wrote:

The fact older branches can still be built hopefully means that the PHP backwards compatibility in MW is systematic (please someone who knows MW correct me if that's wrong).

FWIW, my understanding is that fixes for PHP support in MediaWiki are generally backported to all supported release branches (currently REL1_43/REL1_44/REL1_45). (Unless I'm misunderstanding you & you're referring to something else?)

I found out that:

• I can build old branches such as REL1_39 using the most recent images, which use PHP 8.3 (I did see REL1_42 fail but that seemed an unrelated issue with an extension)
• The tree container images used by the main pod in the mediawiki chart they all get a latest tag
• I could successfully recover env https://568a59f193.catalyst.wmcloud.org/wiki/Main_Page by creating a branched version of the chart the env was built with. Then I modified it to use the latest images and updated the env to use that modified chart

The original issue is fixed, but when trying to rebuild 568a59f193 I ran into this:

Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

Yet another interesting situation. Two patches were pushed in quick succession: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/1240009

Relevant data points:

• https://phabricator.wikimedia.org/T416391#11622065
• https://phabricator.wikimedia.org/T416391#11610925

Another data point. Today the Jenkins Wikilambda CI left a bunch of envs behind:

mw-ext-wl-ci-1238784-24724-4109-py-evaluator-547786dd5c-snpqv   1/1     Running   0               85m
mw-ext-wl-ci-1238784-24724-4109-js-evaluator-97fff64f4-trkbd    1/1     Running   0               85m
mw-ext-wl-ci-1238784-24724-4109-artifact-warehouse              1/1     Running   0               85m
mw-ext-wl-ci-1238784-24724-4109-mariadb-5b4685c7b9-t6qgz        1/1     Running   0               85m
mw-ext-wl-ci-1238784-24724-4109-mediawiki-88494f449-jszph       4/4     Running   0               85m
mw-ext-wl-ci-1239152-82482-4110-js-evaluator-7dd6465b6f-bshbn   1/1     Running   0               79m
mw-ext-wl-ci-1239152-82482-4110-py-evaluator-c756459cd-5njgb    1/1     Running   0               79m
mw-ext-wl-ci-1239344-35683-4111-js-evaluator-7cc5cc6dc4-qfsfd   1/1     Running   0               79m
mw-ext-wl-ci-1239344-35683-4111-py-evaluator-54797b96f7-q75zx   1/1     Running   0               79m
mw-ext-wl-ci-1239152-82482-4110-artifact-warehouse              1/1     Running   0               79m
mw-ext-wl-ci-1239344-35683-4111-artifact-warehouse              1/1     Running   0               79m
mw-ext-wl-ci-1239152-82482-4110-mariadb-689dbdc869-t6dm7        1/1     Running   0               79m
mw-ext-wl-ci-1239344-35683-4111-mariadb-f95cb96cc-tp55n         1/1     Running   0               79m
mw-ext-wl-ci-1239345-10059-4112-py-evaluator-65b68bb7c5-hw8xd   1/1     Running   0               78m
mw-ext-wl-ci-1239345-10059-4112-js-evaluator-69c4755b7b-lqnnw   1/1     Running   0               78m
mw-ext-wl-ci-1239345-10059-4112-artifact-warehouse              1/1     Running   0               78m
mw-ext-wl-ci-1239345-10059-4112-mariadb-68bd8bc87c-6bjzp        1/1     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-py-evaluator-587bc78bc8-gksvs   1/1     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-artifact-warehouse              1/1     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-mariadb-68f895bf85-469ls        1/1     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-js-evaluator-67689db7c7-wpxgn   1/1     Running   0               78m
mw-ext-wl-ci-1239344-62262-4114-js-evaluator-55cc55dff8-kmd6x   1/1     Running   0               77m
mw-ext-wl-ci-1239344-62262-4114-artifact-warehouse              1/1     Running   0               77m
mw-ext-wl-ci-1239344-62262-4114-mariadb-65c79bbb57-k24gs        1/1     Running   0               77m
mw-ext-wl-ci-1239344-62262-4114-py-evaluator-75c4f5cff7-b8778   1/1     Running   0               77m
mw-ext-wl-ci-1239344-35683-4111-mediawiki-6cf876bb58-4vmtz      4/4     Running   0               79m
mw-ext-wl-ci-1239344-62262-4114-mediawiki-5d54bcdcc9-gqmpt      4/4     Running   0               77m
mw-ext-wl-ci-1239152-82482-4110-mediawiki-5fb7ddb8cb-b9jhs      4/4     Running   0               79m
mw-ext-wl-ci-1239345-10059-4112-mediawiki-5c64d87fb7-9x9j6      4/4     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-mediawiki-668b75b6c5-444qf      4/4     Running   0               78m

thank you @Raymond_Ndibe

In T417426#11620327, @taavi wrote:

catalyst-dev currently sees its real usage metrics at about a single CPU core and about 10G of ram, the rest is sitting idle. The last time this came up (T402521) you said you were planning to use these resources to run more workloads in the staging cluster. Did that happen? Why does the staging environment need even more resources when most of that is already idle?

@Raymond_Ndibe We use catalyst-dev to test changes to our infrastructure before rolling those changes out to the production project catalyst, so we generally need the same resources there.

Seems like still happening as of 1.46.0-wmf.15

I caught one in the wild, this time from gitlab: https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-ci-client/-/pipelines/164402

In T416981#11607974, @Jdlrobson-WMF wrote:

There should be no skins in group 2 that are not in group 1 or 0. I'm fairly confident you should be fine tomorrow. Thanks @Tgr for taking care of the Nostalgia issue.

I took another pass through https://codesearch.wmcloud.org/deployed/?q=%5C%5B%27namespaces%27%5C%5D&files=&excludeFiles= and I don't see any code that would trigger new unresolved warnings. It's been hard to catch all the ones so far because it's hard to grep for 'namespaces' in the context of just skins since namespaces are used everywhere with other meanings!

@Urbanecm_WMF thank you for taking care of this

Noise from Nostalgia is gone. Thank you to @Tgr for his patch.

In T416391#11603721, @jeena wrote:

It looks like the environments for wikilambda ci that aren't getting cleaned up are those that did not get a response from catalyst in the appropriate amount of time. When the request to catalyst create times out, then we don't get an environment ID back from catalyst, so even though the post build script has been running, it can't delete the environment.
[...]
See https://integration.wikimedia.org/ci/view/All%20jobs/job/wikilambda-catalyst-end-to-end/1920/console

Unsurprisingly Nostalgia is very much unsupported. I'm pinging the most "recent" (talking about months here) commiters to the repo in case they think they can help: @Reedy @matmarex @Umherirrender

Errors shot up after deploying the train to group1. 3K+ in 15m:

Thanks @Andrew and @bd808 for helping with this!

@Jdforrester-WMF Much obliged again good sir

In T416981#11601720, @Jdforrester-WMF wrote:

All three of these hits are on the WikimediaApiPortal skin — I imagine that skin was missed in the fixes?

Given the likely hit rate on api.wikimedia.org it's probably OK for this new deprecation to not be a train-blocker, as long as it's fixed swiftly?

@Jdforrester-WMF Thank you so much!

Another two separated warnings are also generated per request. For the request ID mentioned above we can see the following in the logs:

Unfortunately it seems the bot-wrangler-traefik-plugin plugin is not compatible with our current K3s production version v1.28.7+k3s1.

A Jenkins upgrade to a new version finally happened as part of this security advisory ticket: T412694. There were no issues reported.