The original issue is fixed, but when trying to rebuild 568a59f193 I ran into this:

Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

Yet another interesting situation. Two patches were pushed in quick succession: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/1240009

Relevant data points:

• https://phabricator.wikimedia.org/T416391#11622065
• https://phabricator.wikimedia.org/T416391#11610925

Another data point. Today the Jenkins Wikilambda CI left a bunch of envs behind:

mw-ext-wl-ci-1238784-24724-4109-py-evaluator-547786dd5c-snpqv   1/1     Running   0               85m
mw-ext-wl-ci-1238784-24724-4109-js-evaluator-97fff64f4-trkbd    1/1     Running   0               85m
mw-ext-wl-ci-1238784-24724-4109-artifact-warehouse              1/1     Running   0               85m
mw-ext-wl-ci-1238784-24724-4109-mariadb-5b4685c7b9-t6qgz        1/1     Running   0               85m
mw-ext-wl-ci-1238784-24724-4109-mediawiki-88494f449-jszph       4/4     Running   0               85m
mw-ext-wl-ci-1239152-82482-4110-js-evaluator-7dd6465b6f-bshbn   1/1     Running   0               79m
mw-ext-wl-ci-1239152-82482-4110-py-evaluator-c756459cd-5njgb    1/1     Running   0               79m
mw-ext-wl-ci-1239344-35683-4111-js-evaluator-7cc5cc6dc4-qfsfd   1/1     Running   0               79m
mw-ext-wl-ci-1239344-35683-4111-py-evaluator-54797b96f7-q75zx   1/1     Running   0               79m
mw-ext-wl-ci-1239152-82482-4110-artifact-warehouse              1/1     Running   0               79m
mw-ext-wl-ci-1239344-35683-4111-artifact-warehouse              1/1     Running   0               79m
mw-ext-wl-ci-1239152-82482-4110-mariadb-689dbdc869-t6dm7        1/1     Running   0               79m
mw-ext-wl-ci-1239344-35683-4111-mariadb-f95cb96cc-tp55n         1/1     Running   0               79m
mw-ext-wl-ci-1239345-10059-4112-py-evaluator-65b68bb7c5-hw8xd   1/1     Running   0               78m
mw-ext-wl-ci-1239345-10059-4112-js-evaluator-69c4755b7b-lqnnw   1/1     Running   0               78m
mw-ext-wl-ci-1239345-10059-4112-artifact-warehouse              1/1     Running   0               78m
mw-ext-wl-ci-1239345-10059-4112-mariadb-68bd8bc87c-6bjzp        1/1     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-py-evaluator-587bc78bc8-gksvs   1/1     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-artifact-warehouse              1/1     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-mariadb-68f895bf85-469ls        1/1     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-js-evaluator-67689db7c7-wpxgn   1/1     Running   0               78m
mw-ext-wl-ci-1239344-62262-4114-js-evaluator-55cc55dff8-kmd6x   1/1     Running   0               77m
mw-ext-wl-ci-1239344-62262-4114-artifact-warehouse              1/1     Running   0               77m
mw-ext-wl-ci-1239344-62262-4114-mariadb-65c79bbb57-k24gs        1/1     Running   0               77m
mw-ext-wl-ci-1239344-62262-4114-py-evaluator-75c4f5cff7-b8778   1/1     Running   0               77m
mw-ext-wl-ci-1239344-35683-4111-mediawiki-6cf876bb58-4vmtz      4/4     Running   0               79m
mw-ext-wl-ci-1239344-62262-4114-mediawiki-5d54bcdcc9-gqmpt      4/4     Running   0               77m
mw-ext-wl-ci-1239152-82482-4110-mediawiki-5fb7ddb8cb-b9jhs      4/4     Running   0               79m
mw-ext-wl-ci-1239345-10059-4112-mediawiki-5c64d87fb7-9x9j6      4/4     Running   0               78m
mw-ext-wl-ci-1239152-59560-4113-mediawiki-668b75b6c5-444qf      4/4     Running   0               78m

thank you @Raymond_Ndibe

In T417426#11620327, @taavi wrote:

catalyst-dev currently sees its real usage metrics at about a single CPU core and about 10G of ram, the rest is sitting idle. The last time this came up (T402521) you said you were planning to use these resources to run more workloads in the staging cluster. Did that happen? Why does the staging environment need even more resources when most of that is already idle?

@Raymond_Ndibe We use catalyst-dev to test changes to our infrastructure before rolling those changes out to the production project catalyst, so we generally need the same resources there.

Seems like still happening as of 1.46.0-wmf.15

I caught one in the wild, this time from gitlab: https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-ci-client/-/pipelines/164402

In T416981#11607974, @Jdlrobson-WMF wrote:

There should be no skins in group 2 that are not in group 1 or 0. I'm fairly confident you should be fine tomorrow. Thanks @Tgr for taking care of the Nostalgia issue.

I took another pass through https://codesearch.wmcloud.org/deployed/?q=%5C%5B%27namespaces%27%5C%5D&files=&excludeFiles= and I don't see any code that would trigger new unresolved warnings. It's been hard to catch all the ones so far because it's hard to grep for 'namespaces' in the context of just skins since namespaces are used everywhere with other meanings!

@Urbanecm_WMF thank you for taking care of this

Noise from Nostalgia is gone. Thank you to @Tgr for his patch.

In T416391#11603721, @jeena wrote:

It looks like the environments for wikilambda ci that aren't getting cleaned up are those that did not get a response from catalyst in the appropriate amount of time. When the request to catalyst create times out, then we don't get an environment ID back from catalyst, so even though the post build script has been running, it can't delete the environment.
[...]
See https://integration.wikimedia.org/ci/view/All%20jobs/job/wikilambda-catalyst-end-to-end/1920/console

Unsurprisingly Nostalgia is very much unsupported. I'm pinging the most "recent" (talking about months here) commiters to the repo in case they think they can help: @Reedy @matmarex @Umherirrender

Errors shot up after deploying the train to group1. 3K+ in 15m:

Thanks @Andrew and @bd808 for helping with this!

@Jdforrester-WMF Much obliged again good sir

In T416981#11601720, @Jdforrester-WMF wrote:

All three of these hits are on the WikimediaApiPortal skin — I imagine that skin was missed in the fixes?

Given the likely hit rate on api.wikimedia.org it's probably OK for this new deprecation to not be a train-blocker, as long as it's fixed swiftly?

@Jdforrester-WMF Thank you so much!

Another two separated warnings are also generated per request. For the request ID mentioned above we can see the following in the logs:

Unfortunately it seems the bot-wrangler-traefik-plugin plugin is not compatible with our current K3s production version v1.28.7+k3s1.

A Jenkins upgrade to a new version finally happened as part of this security advisory ticket: T412694. There were no issues reported.

Production DB migrated

We will announce a maintenance window for the migration.

DB migrated successfully in production

Wikis can be created from Patchdemo in production again

Things have been stable for more than 4h now

Errors are still happening in production. Please don't close unless the fix gets backported or we decide to wait until next week's train

We will announce a maintenance window for the migration.

Warnings no longer reproduce when hitting the affected pages on mediawiki.org. I think it's ok to go ahead and close the task

@ihurbain thanks for the fix!

In T408274#11402879, @Aklapper wrote:

@brennen: I could take over as a backup.

Thank you @Ladsgroup!