From meeting today: maybe use k8s namespaces for this instead to set limits.

A few options. Some of these could be combined:

• CI pipelines deploy 1 pod instead of 5
• Keep only the las N envs created by the func-orch CI pipelines
• Reduce expiry date for CI envs
• Do AW actually use these envs for debugging? Maybe we can unconditionally remove them at the end of the pipeline
• Namespaces: We wouldn't run into the same issues that we had last time for Patchdemo envs
• New dedicated node

I think we want to have two goals here:

1. Reduce overall amount of resources used by CI in a given time window
2. Provide an absolute upper limit for resource usage to prevent CI from affecting other workloads

After chatting with some folks from Abstract Wikipedia, they're not usually using these environments for troubleshooting. And, given the strain on our infra, we're going to unconditionally remove the environments at the end of each test run.

In future, we may be able to provide a way to spin up the same environment, for now, this task is about removing environments at the end of a run.

Change #1184944 had a related patch set uploaded (by Jeena Huneidi; author: Jeena Huneidi):

[integration/config@master] Always clean up catalyst environments

https://gerrit.wikimedia.org/r/1184944

Change #1184944 merged by jenkins-bot:

[integration/config@master] jjb: [wikilambda-catalyst*] Always clean up catalyst environments

https://gerrit.wikimedia.org/r/1184944