To avoid overloading the cluster, we should find a way to disallow ci from creating too many demos.
Some ideas that came up in the meeting:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| set default max envs per token to 10 | repos/test-platform/catalyst/catalyst-api!176 | jnuche | T417304 | main | |
| add support for token-level env limits | repos/test-platform/catalyst/catalyst-api!175 | jnuche | T417304 | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T333548 Write unit/integration tests for Ultraviolet | |||
| Resolved | jeena | T419188 Use Catalyst API to deploy Ultraviolet in CI for E2E testing | |||
| In Progress | BUG REPORT | None | T415952 Intermittent catalyst build failures for wikilambda with ERROR: Environment logs are still not ready. | ||
| Resolved | jnuche | T417304 Put a limit on demos created by ci |
As an interesting note, here's an histogram of Catalyst environment usage by CI over time.
We saw a significant jump earlier this year. As of March 13th, CI pipelines have already created 350 envs this month; roughly 27 a day:
jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/175
add support for token-level env limits
jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/175
add support for token-level env limits
CI pipelines from Abstract Wikipedia now have a limit of 15 10 non-deleted environments (each AW's env takes up 5 pods, allowing 15 envs brings us dangerously close to the pods limit already at our current usage level)
jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/176
set default max envs per token to 10
jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/176
set default max envs per token to 10