Horizon and openstack-browser are showing a usage of 9 instances for deployment-prep in eqiad1-r. It should be more like ~68. I imagine the VCPU/RAM calculations are off too. I believe this data comes from the Nova service itself.
|Open||None||T210215 Quota usage not being counted properly in new region|
|Resolved||bd808||T211718 Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues|
- Mentioned In
- T204688: cloudvps: shiny-r project trusty deprecation
T211718: Total calculated VCPU and RAM numbers for pluggableauth project are incorrect, causing quota issues
- Mentioned Here
- T211393: openstack-browser and horizon: Security group and floating IP quota information being pulled from Nova instead of Neutron for eqiad1-r
Found upstream at https://bugs.launchpad.net/nova/+bug/1742826/comments/4:
There is a well-known issue with quotas "going out of sync" in Nova versions Ocata and earlier and is why the 'nova-manage project quota_usage_refresh' command existed. Quotas out-of-sync means that the quota_usages do not match the actual resources being consumed. This can occur due to races while restarting nova-compute, etc.
I also found https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/refresh-quotas-usage.html and https://blueprints.launchpad.net/nova/+spec/refresh-quotas-usage. I think this "fix" missed mitaka, landed in newton, and was made obsolete in pike, but I'm not 100% certain. I definitely can't find a working nova-manage project quota_usage_refresh or similar on cloudcontrol1003.
It might be worth looking at https://github.com/cernops/nova-quota-sync/blob/master/nova-quota-sync but really I'm holding out hope that this will make more sense in Newton (although the pike fix is the only smart one!)
This script seems to set things right. I've installed it as a standard utility on nova control nodes -- time will tell if this is a one-off fix or if things drift out of sync again immediately.