Thank you @Zabe!

Will backporting https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1248011 unblock the train? I can do that now if it's ready. I see it has a +1.

Yes, that's correct. I remember having a conversation about it so when I saw this task I thought there must be another task for it, but I don't see one.

The timeout period is 2 minutes, which seems generous to me.

It looks like the environments for wikilambda ci that aren't getting cleaned up are those that did not get a response from catalyst in the appropriate amount of time. When the request to catalyst create times out, then we don't get an environment ID back from catalyst, so even though the post build script has been running, it can't delete the environment.

related: https://phabricator.wikimedia.org/T416391

I have archived the documentation page so now we can remove the chart.

I would like to remove the chart from the repo. I'm wondering if the tutorial needs updating or archiving?

@Catrope, I updated our install image to use node 22 and was able to create a demo with a codex patch. Can you confirm the issue is resolved?

Resolving since we used your suggestion when implementing wikifarms in Patch Demo. Thanks!

Since the empty space on the end is taken up during deployment, maybe it would be better like this:

Here is one idea:

CloudPirates mariadb helm chart:
According to https://www.cloudpirates.io/en/about they have been around since 2021. The helm chart was created in August and updated 3 days ago. The CloudPirates helm charts project has 18 contrubutors.

• Mariab offers a kubernetes CRD (https://github.com/mariadb-operator/mariadb-operator), which can be used to define a mariadb instance. It was created in 2022 and is an active project with 66 contributors and the latest release on Aug. 8. Since it’s supported by mariadb itself I think we can put our confidence in its continued maintenance and stability. Releases are well documented with regards to upgrading for breaking changes, which has happened a few times in the past year.

In T386930#11127868, @Michael wrote:

That would be great! Then I don't have to figure out now how to set up all the dependencies. (Though I'd expect that to be somewhat simpler on my Ubuntu than on a Mac?)

In T386930#11125748, @Michael wrote:

Is there a way to test my changes? (A patch-demo for patch-demo?)
I think my change should fix the obvious issues, but maybe there are more problems layered below those.

In T388056#11075208, @vaughnwalters wrote:

Yes thank you, I received both (one from WikiLambda, one from daily run) and they are distinguishable. Will this also allow me to distinguish between orchestrator and (when it lands) evaluator catalyst / selenium failures as well or is that a different scope?

Here's what I think happened based on my testing.

I still haven't been able to reproduce this issue. Based on the creation time of this task it would appear that the expiry date showed up before the third revision, but editing a wiki doesn't change the expiry date at this time.

I ran the expiry checker locally but no demos were affected. I've copied the example demo in production and will wait to see if the expiry date gets changed.

I do see two additional patch requests in the catalyst-api logs, without a subsequest request to stream logs, which means that the expiry date was likely changed at some point. I thought at first the time was close to the expiry checker's run time but after double checking, it doesn't seem to coincide with the expiry checker's run time. If one is in possession of an api key they could also make a request to catalyst to change the expiry date. This happened before the final revision.

[GIN] 2025/07/18 - 12:48:20 | 200 |   1.59035597s |       10.42.0.1 | PATCH    "/api/environments/1362"
[GIN] 2025/07/18 - 12:48:23 | 200 |  102.389774ms |       10.42.0.1 | GET      "/api/environments/1362?latestStatus=0"
[GIN] 2025/07/18 - 12:48:38 | 200 |   84.083174ms |       10.42.0.1 | GET      "/api/environments/1362?latestStatus=0"
[GIN] 2025/07/18 - 12:48:40 | 200 |  1.524394193s |       10.42.0.1 | PATCH    "/api/environments/1362"
[GIN] 2025/07/18 - 12:49:04 | 200 |   1.39868902s |       10.42.0.1 | GET      "/api/environments/1362?latestStatus=1"

@cscott I was not able to reproduce with the existing patches. I also tried with a WIP patch since I see you mentioned it and still couldn't reproduce. I noticed that the example patchdemo has 9 revisions. Can you remember when you first saw the expiry date show up?

Just noting that scap backport --revert can prepare the revert commit and deploy!

I have done the backport and confirmed the changes on cawiki in the recent changes list, confirmed no change to the related changes list, and no changes to English wikipedia. I haven't noticed the error in the logs.

I have deleted the instance and attached volume from the catalyst project in horizon.

Deleted the web proxy as an inital step.

I've switched all the repos in /mnt/k3s-data to use gerrit-replica.wikimedia.org.

Proposed UI sketches