This came during a CI/nodepool outage, OpenStack thought there were over 32,000 servers in the contint cloud project when there were, at the time, less than 10.
Quoting from IRC:
<chasemp> | Servers | 32536 | <chasemp> that's how many servers it thinks are in that project <chasemp> I see 7 in reality <chasemp> that's far more than total instances, that's more like all instances ever created by nodepool since x or something
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T142952 2016-08-10 CI incident follow-ups | |||
| Declined | None | T143018 OpenStack misreports number of instances per project |
Should this be high priority due to it affecting ci nodepool, preventing us from using nodepool until we find and fix most of the problems?
What is the command to get that 32k instances result? The only thing I see approaching is the usage metric:
(openstack) usage show Usage from 2016-07-18 to 2016-08-16 on project contintcloud: +---------------+-------------+ | Field | Value | +---------------+-------------+ | CPU Hours | 9074.97 | | Disk GB-Hours | 181499.43 | | RAM MB-Hours | 18585541.94 | | Servers | 32421 | +---------------+-------------+
Which with the roughly 1k job build per day on Nodepool instance seems consistent (1 month ~> 30 days ~> ~ 30k instances spawned).
Some parts of this are confused by the adhoc nature of reporting on our end, the usage command is indeed cumulative. AFA nodepool and quota and instance management it's hard to separate out specific parts of this. I added details to T143016 so I'm going to close this in favor of that task.
For reference, nova absolute-limits seems to show the quota and their usage:
$ nova absolute-limits +--------------------+-------+--------+ | Name | Used | Max | +--------------------+-------+--------+ | Cores | 20 | 40 | | FloatingIps | 0 | 0 | | ImageMeta | - | 128 | | Instances | 10 | 15 | | Keypairs | - | 100 | | Personality | - | 5 | | Personality Size | - | 10240 | | RAM | 40960 | 102400 | | SecurityGroupRules | - | 20 | | SecurityGroups | 1 | 10 | | Server Meta | - | 128 | | ServerGroupMembers | - | 10 | | ServerGroups | 0 | 10 | +--------------------+-------+--------+