Tagging in @jnuche here since I am wondering if this is related to the recent deployment of T396871: Automatically notify Catalyst when an env deployment's state changes. The reason for my wonderment is:

• It changed today
• the logs show operation not available for environment with id '1584'. Current env status is 'failed' and I remember something about environment status

Interesting, it looks like during the rebuild the environment transitioned to failed temporarily before recovering.

I can't reproduce the issue locally or in prod, it's possible this was a hiccup in the K8s cluster itself. @DLynch would it be ok if I rebuild your wiki-5ad535f7f7 env again?

This also happened to me when rebuilding e34a5ae872 yesterday at around ~18:10 UTC. Attempting to rebuild the wiki for a second time worked.

I've been able to reproduce the issue locally and I think I know what's happening.

The new env watcher adds labels to the environments so they can be tracked, older envs don't have those labels so the API adds them during a rebuild. But it turns out the label change triggers a deployment deletion event from the cluster, which is interpreted by the API as the env being unavailable. Once the new, labelled version of the deployment comes to life, the API detects that and restores the env.

If my theory is right the problem will affect any envs created before we rolled out the env watcher and it will reproduce the first time the env is rebuilt. It's a transitory state, and the environment itself is fine and will recover, but it's a bit scary to see that failure when you first try to rebuild your env.

@A_smart_kitten Can I ask you to try to rebuild one of your old envs and see if the same pattern repeats? A failure followed by a succesful rebuild. In fact, if I'm correct the second rebuild shouldn't even be necessary, the env will eventually recover after a few minutes; simply reloading the Patchdemo UI should show the env running again.

Copying IRC conversation from #wikimedia-releng (log):

2025-09-19 11:10:25  <A_smart_kitten> jnuche: do I understand your "older envs don't have those labels" comment in https://phabricator.wikimedia.org/T405035#11196930 correctly, to mean that this error should only occur when rebuilding an older environment?
2025-09-19 11:11:11 <jnuche> A_smart_kitten: yeah, that's my current working theory
2025-09-19 11:12:27 <jnuche> "older" as in created before yesterday at 13:29 UTC :)
2025-09-19 11:12:29 <A_smart_kitten> jnuche: huh, bc the envionment i rebuilt had only been created initially less than 20mins or so beforehand. (I had to rebuild it to take in a DNM patch I created to enable a couple of config vars, in lieu of T372980 :P)
2025-09-19 11:13:39 <jnuche> ah, I didn't mean to imply old as in "it's been running for a while". Anything created before the time I mention above should be affected
2025-09-19 11:14:41 <jnuche> A_smart_kitten: I think I have a fix, but I'd like to confirm my theory first
2025-09-19 11:15:03 <A_smart_kitten> ah, i didn't see your second reply before i sent mine; but as it happens, the env this happened to me on yesterday was initially created at 2025-09-18 17:56:36
2025-09-19 11:15:33 <A_smart_kitten> according to the patch demo UI, which seems accurate to my memory
2025-09-19 11:15:51 <jnuche> hum, yeah, that wouldn't fit then
2025-09-19 11:17:06 <A_smart_kitten> FWIW i just rebuilt 3387e46cc4 without changing any of the config and I didn't get this error

FWIW though @jnuche, I did reproduce the error just now with f9030db6db by:

• creating a brand-new patch demo wiki with https://gerrit.wikimedia.org/r/1188868/8 attached to it (as opposed to 3387e46cc4, which is patchless), &
• rebuilding it after also adding https://gerrit.wikimedia.org/r/1189543/2 to the env's list of patches.

Thanks @A_smart_kitten, I could reproduce the issue using your steps. I also saw that the env recovered on its own without the need for a second rebuild.

I think my original idea may still be on the right track, even if it's not the full explanation. Something besides envs without labels is triggering transient cluster-side deletion events.

At this point I'm going to add some code to the API to prevent the symptoms and monitor. In the code path that controls the transition of an env to failed during a rebuild, I'm going to add a synchronous check for the env status; I'll also add some logging there. This should prevent the exception being thrown back to the users, whatever the actual root cause is, and give us some logs in the future when it triggers again.

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/135

improve handling of K8s deployment events in the environment watcher

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/135

improve handling of K8s deployment events in the environment watcher

To confirm: I rebuilt the specific patchdemo that was causing reproducible issues for me, and it's definitely fixed for that one.