Page MenuHomePhabricator

Quota increase request for project catalyst-dev
Closed, ResolvedPublic

Description

Project Name: catalyst-dev
Type of quota increase requested: CPU, RAM, and volume storage
Amount to increase:
VCPU: +16
RAM: +32GB
Volume storage: +450GB

Reason:

In T407733, resources were increase for the catalyst project to allow for creation of another worker node. Now we need matching resources in the catalyst-dev project so that we can bring up another worker node there too.

Event Timeline

@dancy I don't know much about catalyst, It'd be interesting to know why we are requesting similar increases as catalyst, for catalyst-dev?
to be clear the question above won't prevent increase if approved by the team, forwarding to irc right now. It's just personal curiosity.

@Raymond_Ndibe We use catalyst-dev to test changes to our infrastructure before rolling those changes out to the production project catalyst, so we generally need the same resources there.

@Raymond_Ndibe We use catalyst-dev to test changes to our infrastructure before rolling those changes out to the production project catalyst, so we generally need the same resources there.

ok got that, thanks @jnuche . Pinging someone again for approval

catalyst-dev currently sees its real usage metrics at about a single CPU core and about 10G of ram, the rest is sitting idle. The last time this came up (T402521) you said you were planning to use these resources to run more workloads in the staging cluster. Did that happen? Why does the staging environment need even more resources when most of that is already idle?

catalyst-dev currently sees its real usage metrics at about a single CPU core and about 10G of ram, the rest is sitting idle. The last time this came up (T402521) you said you were planning to use these resources to run more workloads in the staging cluster. Did that happen? Why does the staging environment need even more resources when most of that is already idle?

It's the same situation as last time. Having the same configuration for both catalyst and catalyst-dev means any testing of configuration changes, migrations, etc that we do on catalyst-dev will be the same that we'll later perform on catalyst, which makes the test more valuable for us. It is already possible to see loaded production workloads in catalyst-dev from the last test we performed there:

jnuche@k3s:~$ hostname -f
k3s.catalyst-dev.eqiad1.wikimedia.cloud
jnuche@k3s:~$ kubectl -n cat-env get po | head
NAME                                                            READY   STATUS             RESTARTS           AGE
envdb-0                                                         1/1     Running            0                  126d
wiki-d9fc83fd11-1876-mediawiki-6bcf9fff89-kxhl6                 2/2     Running            4 (75d ago)        164d
wiki-2f174c3d9a-2274-mediawiki-9f5d96787-wjq4k                  2/2     Running            4 (75d ago)        138d
wiki-e28e437a20-1972-mediawiki-5f698fff58-n8mpx                 2/2     Running            4 (75d ago)        160d
mw-ext-wl-ci-1190984-62361-2175-artifact-warehouse              1/1     Running            2 (75d ago)        145d
wiki-d28ccc16c7-2251-mediawiki-5b7f457cc6-rn9cn                 2/2     Running            4 (75d ago)        139d
wiki-9c858794fb-2106-mediawiki-c58f888db-vd5hx                  2/2     Running            4 (75d ago)        150d
wiki-43b2909e45-2212-mediawiki-6969f6867-gdrsn                  2/2     Running            4 (75d ago)        143d
wiki-27a4ec2259-2089-mediawiki-d8448c56f-fxnzm                  2/2     Running            4 (75d ago)        150d

The same amount of disk is necessary to be able to load the environments.

The same amount of CPU and memory is not necessary but it's useful because it means we are testing the same setup as production when we carry out a test; plus we don't need to maintain two separate infra configurations. But as mentioned last time we are happy to find a compromise if those two resources are a concern: T402521#11111029

Mentioned in SAL (#wikimedia-cloud-feed) [2026-02-16T17:23:38Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase by 16 cores, 450 gigabytes, 32768 ram (T417426)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-02-16T17:23:47Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 16 cores, 450 gigabytes, 32768 ram (T417426)

Before:

raymond-ndibe@cloudcontrol1006:~$ sudo wmcs-openstack quota show catalyst-dev
+-----------------------+-------+
| Resource              | Limit |
+-----------------------+-------+
| cores                 |    40 |
...
| ram                   | 81920 |
...
| gigabytes             |   750 |
...
+-----------------------+-------+

After:

raymond-ndibe@cloudcontrol1006:~$ sudo wmcs-openstack quota show catalyst-dev
+-----------------------+--------+
| Resource              |  Limit |
+-----------------------+--------+
| cores                 |     56 |
...
| ram                   | 114688 |
...
| gigabytes             |   1200 |
...
+-----------------------+--------+
raymond-ndibe@cloudcontrol1006:~$