Page MenuHomePhabricator

Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues
Closed, ResolvedPublic

Assigned To
Authored By
CCicalese_WMF
Dec 11 2018, 8:06 PM
Referenced Files
F27509888: image.png
Dec 12 2018, 4:37 AM
F27509898: image.png
Dec 12 2018, 4:37 AM
F27500423: image.png
Dec 11 2018, 8:06 PM
Tokens
"Evil Spooky Haunted Tree" token, awarded by Krenair.

Description

I'm seeing something rather odd in horizon in my pluggableauth project. The overview says I have exceeded the VPCU and RAM quotas, although if you look at the list of instances, I am below the quotas. I was at the quota until I deleted an instance, at which point the numbers went wonky. I now cannot create a new instance.

image.png (1×1 px, 220 KB)

Event Timeline

bd808 renamed this task from Total calculated VCPU and RAM numbers are incorrect, causing quota issues to Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues.Dec 11 2018, 8:11 PM
bd808 edited projects, added Cloud-VPS; removed Horizon.

Could be. https://tools.wmflabs.org/openstack-browser/project/pluggableauth seems to match what Horizon is showing @CCicalese_WMF:

7 / 8 instances. 14 / 8 VCPUs. 28.0 GB / 16.0 GB RAM. 0 / 0 floating IPs. 1 / 10 security groups.

Those values do not seem to match other checks:

$ OS_REGION_NAME=eqiad1-r OS_PROJECT_ID=pluggableauth openstack server list
+--------------------------------------+----------------------+--------+---------------------------------------+
| ID                                   | Name                 | Status | Networks                              |
+--------------------------------------+----------------------+--------+---------------------------------------+
| 4b3ea8d4-b7e0-4665-a681-51601dae6db4 | pluggableauth-server | ACTIVE | lan-flat-cloudinstances2b=172.16.2.43 |
| 70735f7a-ee7d-435c-828a-b7af800520b3 | cindy                | ACTIVE | lan-flat-cloudinstances2b=172.16.2.42 |
| e8ac63f4-eba6-42c1-bb10-32a3f5ded1f4 | wikifarm             | ACTIVE | lan-flat-cloudinstances2b=172.16.2.41 |
+--------------------------------------+----------------------+--------+---------------------------------------+
$ OS_REGION_NAME=eqiad OS_PROJECT_ID=pluggableauth openstack server list
# empty result

The quotas are tracking 4 "lost" instances somewhere?

I'm pretty sure I haven't lost 4 instances. At least not intentionally. I have deleted instances in the past with no issues. But before today I haven't deleted an instance in over a year, probably - definitely none since the migration.

I see a pattern in the numbers, but it may be a complete red herring:

Before deleting instance this morning:4 instances8 VCPUs16 GB RAM
After deleting instance this morning (expected results):3 instances6 VCPUs12 GB RAM
Add those together (actual wonky results):7 instances14 VCPUs28 GB RAM

While we get this sorted out, is there any chance my quota could get bumped up so I can create the new instance I need? Thanks!

While we get this sorted out, is there any chance my quota could get bumped up so I can create the new instance I need? Thanks!

What size of instance do you still need to build here? I think that you are right that we can unblock you by giving a temporary bump that covers what the system thinks you are using plus the space you need and then reverting that once the instance is created.

I think that you are right that we can unblock you by giving a temporary bump that covers what the system thinks you are using plus the space you need and then reverting that once the instance is created.

Weird unexplained quota bugs are a pretty good reason to do this out of the usual process, yeah :)

One more medium instance would be lovely, thank you :-)

One more medium instance would be lovely, thank you :-)

$ OS_REGION_NAME=eqiad1-r nova quota-update --cores 16 --ram 32768 pluggableauth
$ OS_REGION_NAME=eqiad1-r nova --os-project-id pluggableauth quota-show
+-----------------------------+-------+
| Quota                       | Limit |
+-----------------------------+-------+
| instances                   | 8     |
| cores                       | 16    |
| ram                         | 32768 |
| floating_ips                | 0     |
| fixed_ips                   | 200   |
| metadata_items              | 128   |
| injected_files              | 5     |
| injected_file_content_bytes | 10240 |
| injected_file_path_bytes    | 255   |
| key_pairs                   | 100   |
| security_groups             | 10    |
| security_group_rules        | 20    |
| server_groups               | 10    |
| server_group_members        | 10    |
+-----------------------------+-------+

I think that will give you space to make you medium instance @CCicalese_WMF

I'm getting the same issue:

image.png (920×818 px, 80 KB)

The overview also isn't showing that I have two instances presently:

image.png (382×957 px, 41 KB)

I've been running an installer, testing it, then scrapping the VM and buildng a new one...so I've definitely deleted a few in the last few days.

bd808 claimed this task.

I dropped the quota back down to 8 cores / 16GB. See T210215: Quota usage not being counted properly in new region for future work on the meta issue of quota tracking with the version of OpenStack we are currently running.