+1

For the disk, it's not possible to shrink an existing volume, so the process here would be to create two smaller volumes and move the data from the big to one of the smaller ones, then delete the big one, so the extra space will be needed temporarily.
+1 for that too.

@ssastry can you report back when you have moved the data from the big volume to the small one? That way we can shrink the quota again and free some resources.

Thanks! If you can bump up the volume quota with the other quotas, I can work on the rest and report back.

+1

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:51:49Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T403114)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:51:56Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T403114)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:56:01Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T403114)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:56:08Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T403114)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-03T21:08:42Z] <bd808@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T403114)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-03T21:08:49Z] <bd808@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T403114)

@ssastry was not able to make a 600GB volume even though the quota looked like it was available. He and I both were being capped at making a 499GB volume in the project. I increased the quota by 150GB to get past that accounting bug.

bd808@cloudcumin1001:~$ sudo cookbook wmcs.openstack.quota_increase --gigabytes 150 --project wikitextexp --task-id T403114 --cluster-name eqiad1
Loading WMCS config from /etc/spicerack/wmcs.yaml
START - Cookbook wmcs.openstack.quota_increase
----- OUTPUT of 'sudo -i wmcs-ope...-cloud novaadmin' -----
================
PASS |██████████████████████████████████| 100% (1/1) [00:02<00:00,  2.02s/hosts]
FAIL |                                          |   0% (0/1) [00:02<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'sudo -i wmcs-ope...-cloud novaadmin'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Increased quotas by 150 gigabytes
END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)

Hmmm... still getting this error:

 Error: Unable to create volume. Details
VolumeSizeExceedsLimit: Requested volume size 600G is larger than maximum allowed limit 499G. (HTTP 413) (Request-ID: req-e4f739ff-4f75-4f6a-a4a1-4849dd045ea0)

Did I bump the wrong limit?

Thanks for trying.

For now, I have a good enough workaround. Even though the existing parsing volume attached to ctt-qa-03 on /srv is 551 gb full, i can probably free up some space there by deleting old images for wikis that we've already deployed to and bring it down under 475 gb or so and then back it up before detaching /srv/ on ctt-qa-03 and deleting the volume and attaching a fresh volume with lower disk capacity and restoring from the backup.

Even after I deleted the 1.5TB volume (parsing), looks like I cannot create a new volume larger than 499 GB. Now, this needs fixing. I do need a volume of 1TB, and there is 1.7TB space available in the quota. I can file a new phab task for this if required.

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-04T07:25:12Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.quota_increase (T403114)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-04T07:25:19Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T403114)

In T403114#11145839, @bd808 wrote:

Hmmm... still getting this error:

 Error: Unable to create volume. Details
VolumeSizeExceedsLimit: Requested volume size 600G is larger than maximum allowed limit 499G. (HTTP 413) (Request-ID: req-e4f739ff-4f75-4f6a-a4a1-4849dd045ea0)

Did I bump the wrong limit?

There's indeed two limits, overall volume size and per-volume max size. I just bumped the latter with:

cookbook wmcs.openstack.quota_increase --project wikitex
texp --task-id T403114  --per-volume-gigabytes 1000 --cluster-name eqiad1

Please try again!

Worked! Thanks!

In T403114#11128863, @dcaro wrote:

+1

For the disk, it's not possible to shrink an existing volume, so the process here would be to create two smaller volumes and move the data from the big to one of the smaller ones, then delete the big one, so the extra space will be needed temporarily.
+1 for that too.

@ssastry can you report back when you have moved the data from the big volume to the small one? That way we can shrink the quota again and free some resources.

@dcaro: I have now shuffled around storage, content, and volumes and ctt-qa-03 has 500gb and ctt-prv-04 has 1tb. The rest of the storage can be reclaimed for now. If I need more storage in the future, I'll file a new request. Thanks!

Mentioned in SAL (#wikimedia-cloud) [2025-09-05T14:05:07Z] <dhinus> openstack quota set --gigabytes 1500 wikitextexp (T403114)