Page MenuHomePhabricator

Consider different varieties of Cloud VPS instance flavors
Open, LowestPublic

Description

Right now our standard VPS offerings are m1 small, medium, large, and xlarge. They vary by three attributes -- RAM, storage, and CPU -- and they all rise as you go up the scale. This assumes, for instance, that because someone has large RAM needs, they also have large storage and large processing needs. This isn't necessarily true.

As we want to encourage responsible resource use and not have people provision resources they don't need, we should consider additional types of instances that scale resource allocation differently. I think DigitalOcean has a useful model here -- you can get "standard" instances comparable to our current offerings, but you can also get optimized instances that offer less storage in favor of more/better RAM and CPU. (Our bigram flavor is based on the same principle but was created ad hoc to fulfill a particular need.)

I am posting this as an idea. No urgent action is needed.

Event Timeline

Harej triaged this task as Low priority.Mar 5 2018, 7:46 PM
Harej updated the task description. (Show Details)

Other flavors I heard of / used: gigantic (16 CPU, 16G RAM, 80G disk), moregigantic (16 CPU, 16G RAM, 160G disk), bigdisk (4? CPU, 24G RAM, 300G disk).

That's very interesting. I didn't know about gigantic, moregigantic, or bigdisk. Are they flavors we've discontinued? Or are they hidden by default? For what it's worth, bigram still shows up on my menu in Horizon even though I lack quota to use it.

Incidentally I've had this idea for some time to offer very high project quotas and very high performance VPS flavors to match. These would be special cases that would require approval and would be reserved for particular high-impact projects. Sort of like a grants program where you make a business case for your work.

  • gigantic is in use by the video project for transcoding.
  • moregigantic was used in the same project for a single transcoding instance with a larger disk space (T129581), but we found it unnecessary after we moved 'transcoded files' to NFS in T152899
  • bigdisk is from T169133, but openstack-browser now reports it having UNKNOWN flavor. Another use I found is T178332, but it now shows justdisk

I don't know whether they are hidden or discontinued, as I do not manage any projects mentioned. (v2c is just the video project's biggest consumer :) )

There was, according to my history-digging a long time ago (because I got curious about flavors), and if I remember correctly, flavors with another prefix (like the m1 now).

Related tasks:

  • T132194 (which just plainly proposes new flavors without recommending what those flavors might be)
  • T95731 (which specifically requests a large-storage instance; I think the long term solution for that will be detachable block storage)
  • T159930 (also a request for a large-storage instance)

Incidentally I've had this idea for some time to offer very high project quotas and very high performance VPS flavors to match. These would be special cases that would require approval and would be reserved for particular high-impact projects. Sort of like a grants program where you make a business case for your work.

We do that, but we do it on a one-off basis and don't advertise it. Everybody thinks their project should have all the resources. :)

Since the default disk allocation for all VMs is the same 20G used by the base image and additional disk is only used when provisioned via Puppet, there isn't a huge amount of utility in these more RAM + CPU & less disk types. Quota is done by RAM and CPU with disk utilization being incidentally capped just by the flavors that we offer. If projects have exotic needs I think it would generally be better for them to talk to the cloud-services-team in a project/quota request than to offer a wide array of self-service options.

Since the default disk allocation for all VMs is the same 20G used by the base image and additional disk is only used when provisioned via Puppet

This is not apparent to the end-user. When I see a flavor with 80 GB of storage I expect that it will come with 80 GB of storage, not 20 GB with an additional 60 GB volume made available through /srv. I've had to assure people that we weren't cutting their disk quota, just that they needed to set up the extra storage. We should advertise each flavor as having the same storage capacity, with the option to define additional storage capacity.

This also doesn't address the inverse situation, where someone needs a lot of storage but not much computing power. In that case, they are forced to go with the flavor with more computing power than they need; they can't just not use the extra RAM/CPU.

Since the default disk allocation for all VMs is the same 20G used by the base image and additional disk is only used when provisioned via Puppet

This is not apparent to the end-user. When I see a flavor with 80 GB of storage I expect that it will come with 80 GB of storage, not 20 GB with an additional 60 GB volume made available through /srv. I've had to assure people that we weren't cutting their disk quota, just that they needed to set up the extra storage. We should advertise each flavor as having the same storage capacity, with the option to define additional storage capacity.

The UX for this could be improved, but until we have quota'ed attachable block storage (FY19/20 at the earliest) the "option to define additional storage capacity" pretty much doesn't exist in the OpenStack infrastructure.

This also doesn't address the inverse situation, where someone needs a lot of storage but not much computing power. In that case, they are forced to go with the flavor with more computing power than they need; they can't just not use the extra RAM/CPU.

They are not forced, they can easily ask for and receive a custom image type. If it turns out that more than N projects have a real use for the image type then we could make the option more widely available. Again, I think the major deficiency here is documentation and communication rather than functionality. We are not aiming to be and AWS or Rackspace competitor. We really do want to know what folks are doing in the VMs and why so we can help them make the best use of the shared resource pool which will always be smaller than the global demand.

It's good that we don't have to "fix" this by tomorrow. We can wait for a better solution to come along; in this case, having storage that is separate from computation.

In the meantime, I do think the UI needs to make clearer that all instances start off with 20 GB of storage, and that extra storage is allocated as a separate step.

bd808 lowered the priority of this task from Low to Lowest.Mar 11 2018, 12:09 AM

Just so this doesn't get lost:

1Name t206636: 32 VCPUs, 132 GB RAM, 3481 GB disk space total. Available to projects: [u'wikidata-query']
2Name bigdisk2: 4 VCPUs, 24 GB RAM, 300 GB disk space total. Available to all
3Name parsingtest: 12 VCPUs, 32 GB RAM, 400 GB disk space total. Available to projects: [u'wikitextexp']
4Name bigram: 8 VCPUs, 36 GB RAM, 80 GB disk space total. Available to all
5Name justdisk: 4 VCPUs, 8 GB RAM, 300 GB disk space total. Available to projects: [u'cyberbot']
6Name mediumdb: 16 VCPUs, 64 GB RAM, 1638 GB disk space total. Available to projects: [u'clouddb-services']
7Name mediumram: 8 VCPUs, 24 GB RAM, 80 GB disk space total. Available to projects: [u'integration']
8Name analytics-hadoop-worker: 46 VCPUs, 120 GB RAM, 22000 GB disk space total. Available to projects: [u'cloud-analytics']
9Name bigdisk: 4 VCPUs, 24 GB RAM, 300 GB disk space total. Available to projects: [u'download', u'ign2commons', u'observer', u'wikibrain', u'wikidata-query']
10Name largedb: 16 VCPUs, 64 GB RAM, 3481 GB disk space total. Available to projects: [u'clouddb-services']
11Name bigdisk3: 4 VCPUs, 24 GB RAM, 300 GB disk space total. Available to projects: [u'wikibrain']
12Name m1.medium: 2 VCPUs, 4 GB RAM, 40 GB disk space total. Available to all
13Name m1.small: 1 VCPUs, 2 GB RAM, 20 GB disk space total. Available to all
14Name m1.xlarge: 8 VCPUs, 16 GB RAM, 160 GB disk space total. Available to all
15Name m1.large: 4 VCPUs, 8 GB RAM, 80 GB disk space total. Available to all
16Name c8.m8.s60: 8 VCPUs, 8 GB RAM, 60 GB disk space total. Available to projects: [u'deployment-prep']
17Name m1.gigantic: 16 VCPUs, 16 GB RAM, 80 GB disk space total. Available to projects: [u'video']
18Name dumps-temporary-file-storage: 1 VCPUs, 2 GB RAM, 500 GB disk space total. Available to projects: [u'dumps']
19Name ci1.medium: 2 VCPUs, 2 GB RAM, 40 GB disk space total. Available to projects: [u'integration']
20Name xlarge-xtradisk: 8 VCPUs, 16 GB RAM, 300 GB disk space total. Available to projects: [u'mwoffliner']
21Name c1.m2.s80: 1 VCPUs, 2 GB RAM, 80 GB disk space total. Available to projects: [u'deployment-prep']