In T418521#11807710, @jnuche wrote:

@Dzahn Would it make sense to wait until @hashar is back from his vacation before we do the switch? He's the only one who really knows the Jenkins contint setup

@Dzahn Would it make sense to wait until @hashar is back from his vacation before we do the switch? He's the only one who really knows the Jenkins contint setup

As a first step, I'll test locally and on catalyst-dev how risky the v1.28 -> v1.31 jump is

I can't reproduce the issue on staging anymore after my changes. No error during the creation of ~20 envs.

Tested on catalyst-dev:

• Created a new pod "A"
• Used the new service to create backups on all hosts
• Created a new pod "B"
• Messed up the cluster by deleting several cluster directories on two of the nodes, including /mnt/k3s-data/k3s/data and /mnt/k3s-data/k3s/server on the primary host
• Stopped all K3s systemd services across the cluster
• Rsync'd back from the backups on all hosts
• Restarted systemd services
• Cluster is healthy: Pod "A" is back and pod "B" is not

1. We could try temporarily reverting (at least group1 and group2) to php-1.46.0-wmf.21 to confirm the correlation described above. This may of course end badly if anyone has already assumed .22 is rollback-safe and made dependent changes.

That could be problematic at this point. A quick glance already shows backports for 1.46.0-wmf.22, e.g.: https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/1267214 or https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1268281. The latter fixing a mobile login bug T422320

@Sfaci @cjming In the last two hours there where 2 big bursts of >15K each.

I've backported the fix. Thank you for the patch @matmarex

Tests now passed:

13:02:05 Started check-testservers
13:02:05 Executing check 'check_testservers_k8s-1_of_2'
13:02:05 Executing check 'check_testservers_k8s-2_of_2'
13:02:28 Finished check-testservers (duration: 00m 22s)

Thank you!

In T421988#11777517, @SomeRandomDeveloper wrote:

(AFAICS the tests live in the puppet repo as well: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/abc1f4f57135615c2aef77c793f34a21c58fd199/modules/profile/files/httpbb/appserver)

In T421988#11777426, @matmarex wrote:

It was an intended change. Can you point me to where these tests live, so I can update them?

However, note that the header for es.wikibooks.org seems to be an absolute URL missing the schema

That's expected, it's a protocol-relative URL. We only serve wikis over HTTPS these days, but we still have a lot of legacy configs from back in the day when we supported HTTP too.

In T421828#11773269, @cjming wrote:

backported UBN fix to wmf.22 and it appears errors are dropping 😌

In the last hour just for the test wikis:

• mwversion: 1.46.0-wmf.22
• timestamp: 2026-03-31T07:54:04.828Z
• phpversion: 8.3.30
• reqId: 95100eec-b69f-4dc3-8c79-e3d42a5124ae
• Find reqId in Logstash

• mwversion: 1.46.0-wmf.22
• timestamp: 2026-03-31T07:50:02.298Z
• phpversion: 8.3.30
• reqId: 2d0e0e52-7f9c-4a9b-8e06-cca6d99b99b4
• Find reqId in Logstash

https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/312

Fix is in production

@jnuche would you be able to review the suggested change?

The team went with the MariaDB operator as a replacement.

Possibly related to T419092

Disk quotas for both catalyst and catalyst-dev will need to be raised by 760GB from the current 1200GB to a total of 1520GB

CI pipelines from Abstract Wikipedia now have a limit of 15 non-deleted environments.

As an interesting note, here's an histogram of Catalyst environment usage by CI over time.

All of our MariaDB databases have been now migrated away from bitnami

We know from T405224 that we can recover the entire cluster from the K3s data volumes. Additionally, the folks over at Cloud have told us in the past that it's not possible to automate/schedule the creation of OpenStack snapshots natively.

As usual, we will announce a maintenance window for the migration.

Operation times in production have significantly improved and gone back to the performance levels we had by the end of November 2025.

Similarly to what happened with secrets, recreated envs have been leaving behind a significant number of replica sets behind:

kubectl -n cat-env get rs | grep 750c4d946d
wiki-750c4d946d-3895-mediawiki-66bccd4d77   0         0         0       15d
wiki-750c4d946d-3895-mediawiki-8669b898df   0         0         0       15d
wiki-750c4d946d-3895-mediawiki-668466c6b8   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-c556c45c5    0         0         0       14d
wiki-750c4d946d-3895-mediawiki-76c5fb6d87   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-6968d99c4d   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-6b5b86ff6f   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-68cd949987   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-75749d974b   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-5d46bb4dc5   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-68c5994db9   1         1         1       11d

Some of the helm commands had started slowing to a crawl; as it turns out we had 912 secrets in the cat-env namespace. A majority of those were originating in helm history revisions. A lot of those revisions could be safely removed and after doing that helm commands became noticeably more responsive again. Plus env creation times have gone down again to the levels we had a few months ago: