I looked into both instances and have an explanation as to this round of issues :)

As part of T143938 I bumped up the nodepool load /a small bit/ with https://gerrit.wikimedia.org/r/#/c/307526/. Previously we had some artificial deflation here with quota management issues addressed with https://gerrit.wikimedia.org/r/#/c/306683/. Both of these patches mean nodepool is actually doing close to what it was assumed it was doing before them.

If I look at nova-conductor.log I see a lot of operations like:

2016-09-01 14:50:08.792 11200 INFO nova.db.sqlalchemy.api [req-377cce23-4e54-4e0f-b483-98f2288cb995 nodepoolmanager contintcloud - - -] quota_usages out of sync, updating. project_id: contintcloud, user_id: None, resource: fixed_ips, tracked usage: 26, actual usage: 30

I think with nodepool now using closer to its actual max-servers pool and quota adjustments we are using more of our db connection pool. These limits are pretty low by default at pool size of 5, max-overflow of 10, and a timeout of 30s for getting the connection count to saturation.

Results in errors like:

fault	{"message": "Remote error: RemoteError Remote error: TimeoutError QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30

Which I see in the case of tools-static-04

sqlalchemy pooling
http://docs.sqlalchemy.org/en/latest/core/pooling.html

nova.conf options for addressing this
http://docs.openstack.org/liberty/config-reference/content/list-of-compute-config-options.html#config_table_nova_conductor

Saturation of this pool can result in operations being dropped entirely. In this case there was no actual instance created so a reboot of the entry always resulted in an ERROR state. The compute node had no record of it, etc.

nova.conf does a few odd things here where

min_pool_size appears to not be an option at all in reality.
max_pool_size is just pool_size for sqlalchemy

I think we have been seeing this intermittently for awhile and mistaking it to rabbitmq shenanigans.

Change 307956 had a related patch set uploaded (by Rush):
openstack: tune sqlalchemy pooling behavior

https://gerrit.wikimedia.org/r/307956

Change 307956 merged by Rush:
openstack: tune sqlalchemy pooling behavior

https://gerrit.wikimedia.org/r/307956