On 03/03/17 nodepool stopped working, which is most likely node pool stopped working, but could have been caused by it keeping leaking instances.
All these issues may be bugs which have been fixed in a newer release. We are using a very ancient version of nodepool.
we merged https://gerrit.wikimedia.org/r/#/c/340986/ causing nova services to restart and a host of in-flight instances to go error and some labvirts are coming back slowly. Hopefully, it's all transient. @Andrew is babysitting this now to ensure.e
Mentioned in SAL (#wikimedia-operations) [2017-03-03T17:34:59Z] <hashar> CI is mostly recovered. It could not spawn instance anymore. The queue is being processed and will take a while to be completed. Check status on https://integration.wikimedia.org/zuul/ | T159543
Nova / OpenStack recovered. Thus instances managed to get deleted and Nodepool has then been able to refill the pool with fresh instances.