Request volume quota increase for integration Cloud VPS project
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Mar 17 2022, 2:27 PM

Description

Project Name: integration

Type of quota increase requested: volume quota increase

Reason:

The integration-castor03 is based on Debian Stretch and has to be replaced with a Bullseye image (T252071). It uses a custom flavor to get extra disk space which is no more available via Horizon, we thus a new flavor to be created. That is also an opportunity to revisit its sizing.

We currently have:
g2.cores8.ram16.disk160

VCPUs	8
RAM	16G
Disk	160G

Looking at the Grafana board for integration-castor03:

there is barely any CPU usage, so we can keep 8 vCPUs
RAM is filed by disk cache. I think we would benefit more RAM for that
the partition holding the cache is 138G and is full. Beside we are not too good at garbage collecting the cache, we also have more repositories and jobs and would need an extension
we could use a bump of IO throttling

We would need to create a new instance and offload the disk storage to a volume set to 320G (double the current size). The integration project currently uses 53G out of a 80G disk quota. May you increase the volume quota from 80G to 400G?

Details

	Subject	Repo	Branch	Lines +/-
	ci: manage cinder volume on Castor instance	operations/puppet	production	+8 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	hashar	T292729 TAR_ENTRY_ERROR ENOSPC: no space left on device
Resolved	hashar	T252071 Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye
Resolved	Andrew	T304080 Request volume quota increase for integration Cloud VPS project

Event Timeline

hashar created this task.Mar 17 2022, 2:27 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 17 2022, 2:27 PM

hashar mentioned this in T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye.Mar 17 2022, 2:28 PM

hashar added a project: Release-Engineering-Team.

hashar added a parent task: T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye.Mar 22 2022, 7:43 PM

For this specific use case we are probable better off creating a volume via Horizon and attach it to the instance. That gives us the liberty to resize however we need and saves WMCS people from having to create yet another custom flavor.

The integration has a volume quota though: Used 53GB of 80GB. May we get that one bumped to 400GB?

hashar renamed this task from Request new flavor for integration Cloud VPS project to Request volume quota increase for integration Cloud VPS project.Mar 28 2022, 9:48 AM

hashar updated the task description. (Show Details)

+1 from me

done!

That worked like a charm, I have followed https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_instances#Cinder:_Attachable_Block_Storage_for_Cloud_VPS and ensured labs_lvm is not applied.

Thank you!

As a followup, I had to recreate the instance and reattach the volume but it was not magically mounted to /srv. Andrew said prepare-cinder-volume creates an fstab entry but since I have recreated the instance it was gone.

I thus attached the volume to the instance and applied profile::labs::cindermount::srv.

Notice: /Stage[main]/Profile::Labs::Cindermount::Srv/Cinderutils::Ensure[cinder_on_srv]/Exec[prepare_cinder_volume_/srv]/returns: executed successfully

$ df -h  /srv
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        314G  205G   94G  69% /srv

Magic.

Change 961844 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] ci: manage cinder volume on Castor instance

https://gerrit.wikimedia.org/r/961844

gerritbot added a project: Patch-For-Review.Sep 28 2023, 3:41 PM

Change 961844 merged by David Caro: