Hello!

We are not yet equipped to provide 5-10 TB of storage in one go, especially on top of the 2.5 that are currently allocated for the project. That said, we do want to encourage you to move your existing local instance storage off of VMs and onto cinder volumes. I see that you currently have five hosts each with 500Gb of space allocated per host (but only 1cpu each), so here is what I propose:

• I'll set your current Cinder quota to 2 TB and 10 volumes

• You can migrate your current workload onto skinny VMs (presumably size g2.cores1.ram2.disk20) and experiment with moving your existing data onto Cinder volumes. (If it's transient data that should be totally straightforward; if you need to preserve existing data then I'd advise attaching the volume to the old VM, copying, and then moving the volume to the new VM)

• Once some/most of your data and workload is moved to small VMs + Cinder, you can delete the old 'dumps-temporary-file-storage' flavor instances, and ping me on this ticket with notes about what you've freed up

• at which point I can boost the quota again to take up the space freed be deletion of the -file-storage VMs

• (repeat if necessary)

• finally once you're fully cinderized I'll remove the dumps-teporary-file-storage flavor

Does that sound workable?

I think it is a good plan, let's go ahead with it.

There is no data to be preserved, so I can go ahead to delete and recreate the instances, then attach the Cinder storage.

Also to clarify, in T159930 you mentioned that "it would be moderately useful if you delete/recreate your VMs" on a regular basis, mainly due to the file system that Cloud VPS is using. Can I confirm if this is something that we need to do for the Cinder volumes as well?

I've raised this quota to 2048 GB.

Regarding refreshing cinder volumes -- the same is likely to apply here. If your workflow includes deleting volumes when you're done and later creating fresh ones it will result in more efficient space reclamation. I wouldn't worry about it too much but if it fits into your plans it will safe some space. Thanks.

Thanks! I tried to attach one of the Cinder volumes to dumps-1 and it seems to be working. However, it seems like it is not possible to attach a second Cinder volume to another instance, is it possible to look into this issue?

I was able to reproduce the issue -- it looked like some kind of timeout in the cinder backend. I restarted services and it's working now -- go ahead and try again and we'll see if it degrades again over time.

I got the second volume to attach successfully and everything seems to be working fine so far, except that I/O seems to be slightly slower than using instance storage (but if I'm not wrong they should be writing to the same Ceph storage?).

Nonetheless, I think we can go ahead to raise the quota further as I get the remaining 3 instances migrated to using Cinder storage.

Also to add, the quota for the number of volumes seems to only be updated after you restarted the services. Before that, it always indicated a quota of 4 volumes even when the quota is indeed at 2048 GiB.

I've boosted the quota to 3TB (which should be 512GB more than you were using previously). If you outgrow that you can create a new quota task and we'll consider.

I just double-checked the horizon UI and it's showing the new quota now so I think you should be good to go. I'll do some other testing around the volume quota.

Thanks! Ideally we can work with 5 TB as the size of the dumps are getting bigger (enwiki is now at 1.6 TB, but was at around only 1 TB when T159930 was filed), so I really hope if the Cloud Services team would consider giving us this quota. Again, the space used should hover around 10 GB most of the time, the large space is to handle the situation when large wikis like enwiki or wikidatawiki are being archived.

Otherwise, would it be possible for us to keep the dumps-teporary-file-storage flavor? This allows us to spin up temporary instances in situations where the archiving tasks become backlogged, and can be deleted once the backlog goes down. This keeps our workflow flexible while ensuring that we are not using too much Cloud VPS resources.