Page MenuHomePhabricator
Create Task
Maniphest T402521

Quota increase request for catalyst-dev
Closed, ResolvedPublic
Actions

Description

Project Name: catalyst-dev
Type of quota increase requested: cpu, ram & disk
Amount to increase: 32 cpus, 64 GB ram & 670 GB disk
Reason:

The Catalyst project is currently building an OpenTofu stack to automate the deployment and change management of our Kubernetes cluster and related infrastructure.

We want to use catalyst-dev to reproduce the production environment in order to test data backups, upgrades and more. We're considering to eventually use the project as a staging environment for changes to our infrastructure.

In order to do the above we need to scale up resources to match production infra in the catalyst project

Related Objects

Event Timeline

jnuche created this task.Aug 21 2025, 1:13 PM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptAug 21 2025, 1:13 PM
taavi subscribed.Aug 21 2025, 1:22 PM

Why does the staging environment need to match the production one in resources? I would assume staging would be much less heavily used than the production one.

jnuche added a comment.Aug 21 2025, 1:28 PM
In T402521#11106391, @taavi wrote:

Why does the staging environment need to match the production one in resources? I would assume staging would be much less heavily used than the production one.

Ideally we want to use the same OpenTofu configuration for both the staging and production environments. That would give us more confidence when testing configuration changes in catalyst-dev.

Less ideally we could use different configs for the two projects, with less CPU and memory for catalyst-dev. In that case we would still want the same amount of disk as production to test things like recovery via backups.

• thcipriani subscribed.Aug 22 2025, 3:34 PM

The idea behind the quota increase is to use our real workloads within our staging.

The thinking behind parity is to pull over our existing k8s PVs/PVCs to staging, spin them up, then test changes/migrations/backup restore, etc. within, effectively, production (but without making users sad)—like a blue/green that we never swap, but can reset if something goes awry.

CPU is less important, environments will be slow, but that's fine.

Is that helpful context @taavi? Happy to chat more to find a compromise if a particular resource is a worry for you all.

Andrew subscribed.Aug 27 2025, 2:54 PM

This seems fine to me, obviously if you're able to use smaller VMs please do :)

+1

Raymond_Ndibe claimed this task.Aug 27 2025, 4:30 PM
Stashbot added a comment.Aug 27 2025, 4:49 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-27T16:49:46Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T402521)

Stashbot added a comment.Aug 27 2025, 4:49 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-27T16:49:55Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T402521)

Raymond_Ndibe added a comment.Aug 27 2025, 4:52 PM

before

raymond-ndibe@cloudcontrol1006:~$ sudo wmcs-openstack quota show catalyst-dev
+-----------------------+-------+
| Resource              | Limit |
+-----------------------+-------+
| cores                 |     8 |
| ram                   | 16384 |
| gigabytes             |    80 |
...
+-----------------------+-------+

after

raymond-ndibe@cloudcontrol1006:~$ sudo wmcs-openstack quota show catalyst-dev
+-----------------------+-------+
| Resource              | Limit |
+-----------------------+-------+
| cores                 |    32 |
| ram                   | 65536 |
| gigabytes             |   670 |
...
+-----------------------+-------+
Raymond_Ndibe closed this task as Resolved.Aug 27 2025, 4:53 PM
jnuche reopened this task as Open.Aug 28 2025, 8:13 AM

Thank you for your support with this!

Unfortunately the quotas were not increased by 32 CPUs, 64GB mem and 670GB disk; they were set to those values instead. The total values we need to match production are 40 CPUs, 80GB mem and 750GB disk

Stashbot added a comment.Aug 28 2025, 8:20 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-28T08:20:01Z] <wmbot~dcaro@acme> START - Cookbook wmcs.openstack.quota_increase (T402521)

Stashbot added a comment.Aug 28 2025, 8:20 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-28T08:20:09Z] <wmbot~dcaro@acme> END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) (T402521)

Stashbot added a comment.Aug 28 2025, 8:20 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-28T08:20:13Z] <wmbot~dcaro@acme> START - Cookbook wmcs.openstack.quota_increase (T402521)

Stashbot added a comment.Aug 28 2025, 8:20 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-28T08:20:20Z] <wmbot~dcaro@acme> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T402521)

dcaro closed this task as Resolved.Aug 28 2025, 8:22 AM
dcaro subscribed.

Oops, xd, there you go:

root@cloudcontrol1006:~# sudo wmcs-openstack quota show catalyst-dev | grep ' \(cores\|gigabytes\|ram\) '
| cores                 |    40 |
| ram                   | 81920 |
| gigabytes             |   750 |
jnuche added a comment.Aug 28 2025, 8:27 AM

Thanks a mil :)

taavi mentioned this in T417426: Quota increase request for project catalyst-dev.Feb 16 2026, 2:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits