I'm seeing something rather odd in horizon in my pluggableauth project. The overview says I have exceeded the VPCU and RAM quotas, although if you look at the list of instances, I am below the quotas. I was at the quota until I deleted an instance, at which point the numbers went wonky. I now cannot create a new instance.
Description
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | Andrew | T210215 Quota usage not being counted properly in new region | |||
Resolved | bd808 | T211718 Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues |
Event Timeline
Could be. https://tools.wmflabs.org/openstack-browser/project/pluggableauth seems to match what Horizon is showing @CCicalese_WMF:
7 / 8 instances. 14 / 8 VCPUs. 28.0 GB / 16.0 GB RAM. 0 / 0 floating IPs. 1 / 10 security groups.
Those values do not seem to match other checks:
$ OS_REGION_NAME=eqiad1-r OS_PROJECT_ID=pluggableauth openstack server list +--------------------------------------+----------------------+--------+---------------------------------------+ | ID | Name | Status | Networks | +--------------------------------------+----------------------+--------+---------------------------------------+ | 4b3ea8d4-b7e0-4665-a681-51601dae6db4 | pluggableauth-server | ACTIVE | lan-flat-cloudinstances2b=172.16.2.43 | | 70735f7a-ee7d-435c-828a-b7af800520b3 | cindy | ACTIVE | lan-flat-cloudinstances2b=172.16.2.42 | | e8ac63f4-eba6-42c1-bb10-32a3f5ded1f4 | wikifarm | ACTIVE | lan-flat-cloudinstances2b=172.16.2.41 | +--------------------------------------+----------------------+--------+---------------------------------------+ $ OS_REGION_NAME=eqiad OS_PROJECT_ID=pluggableauth openstack server list # empty result
The quotas are tracking 4 "lost" instances somewhere?
I'm pretty sure I haven't lost 4 instances. At least not intentionally. I have deleted instances in the past with no issues. But before today I haven't deleted an instance in over a year, probably - definitely none since the migration.
I see a pattern in the numbers, but it may be a complete red herring:
Before deleting instance this morning: | 4 instances | 8 VCPUs | 16 GB RAM |
After deleting instance this morning (expected results): | 3 instances | 6 VCPUs | 12 GB RAM |
Add those together (actual wonky results): | 7 instances | 14 VCPUs | 28 GB RAM |
While we get this sorted out, is there any chance my quota could get bumped up so I can create the new instance I need? Thanks!
What size of instance do you still need to build here? I think that you are right that we can unblock you by giving a temporary bump that covers what the system thinks you are using plus the space you need and then reverting that once the instance is created.
Weird unexplained quota bugs are a pretty good reason to do this out of the usual process, yeah :)
$ OS_REGION_NAME=eqiad1-r nova quota-update --cores 16 --ram 32768 pluggableauth $ OS_REGION_NAME=eqiad1-r nova --os-project-id pluggableauth quota-show +-----------------------------+-------+ | Quota | Limit | +-----------------------------+-------+ | instances | 8 | | cores | 16 | | ram | 32768 | | floating_ips | 0 | | fixed_ips | 200 | | metadata_items | 128 | | injected_files | 5 | | injected_file_content_bytes | 10240 | | injected_file_path_bytes | 255 | | key_pairs | 100 | | security_groups | 10 | | security_group_rules | 20 | | server_groups | 10 | | server_group_members | 10 | +-----------------------------+-------+
I think that will give you space to make you medium instance @CCicalese_WMF
I'm getting the same issue:
The overview also isn't showing that I have two instances presently:
I've been running an installer, testing it, then scrapping the VM and buildng a new one...so I've definitely deleted a few in the last few days.
I dropped the quota back down to 8 cores / 16GB. See T210215: Quota usage not being counted properly in new region for future work on the meta issue of quota tracking with the version of OpenStack we are currently running.