From T70113#2479567:

Maybe also look at the median time a change stay in the Zuul queues. In Graphite that would be for example sumSeries(zuul.pipeline.*.resident_time.median). Over the last past 5 weeks:

@hashar where on that is the reduction in concurrent jobs for nodepool? I don't see anything there that screams 'turn me back up!'

Inspired by the graph above, I created a Graph panel at https://grafana.wikimedia.org/dashboard/db/releng-kpis?panelId=5&fullscreen

It reports the time a job waits from when it is known to Zuul until it is starts running. The filter restricts it to Jenkins labels ci-* which are the ones used for Nodepool instances.

The spikes are the conjonction of San Francisco waking up, Europe being active, the SWAT and deploys slot. That should correlate with the number of jobs in Gearman waiting (though that is for any job including Zuul internal jobs) and the max time to launch an instance https://grafana.wikimedia.org/dashboard/db/releng-zuul?panelId=18&fullscreen

The first Graph based on a moving average of one hour might be a good high level representation of the perceived response time. I have asked Release-Engineering-Team internal mailing list for more feedback / tuning.

The https://grafana.wikimedia.org/dashboard/db/releng-zuul?panelId=18&fullscreen second graph is on a per label basis.

For reference, the Graphite query for the first Graph is:

alias(movingAverage(consolidateBy(maxSeries(zuul.pipeline.*.label.ci-*.wait_time.upper), 'max'), '1hour'), 'One hour moving average')

alias(
     movingAverage(
          consolidateBy(
               maxSeries( zuul.pipeline.*.label.ci-*.wait_time.upper),
          'max'),
     '1hour'),
'One hour moving average'
)

In T139771#2492434, @chasemp wrote:

@hashar where on that is the reduction in concurrent jobs for nodepool? I don't see anything there that screams 'turn me back up!'

Any spike on the red graphs shows a job waiting minutes for a node to spawn and be pooled in Jenkins. Some are nearly instants because a node is available in the pool and consumed right away, some have to wait for an instance to spawn which takes roughly a minute all included. Anything above that minute is the pool being starved.

We have stopped migrating jobs from the permanent slaves to the Nodepool ones pending capacity.

Another thing we are missing is to have Nodepool to report metrics to statsd (T111496) specially the number of instances per state (eg: 5 ready, 3 building, 2 deleting). That will give you a better view of the state of the pool.

We had an outage for CI 2 night ago and during that we discovered that nodepool seems to be waiting only 1s before declaring build on a VM faulty and then issuing a delete and then eventually churning on its own quota limitations. This happened because we upped the timeout allowance for instance creation as we have larger and larger projects with relative rule sets. During debugging we also discovered issues with quota tracking and nodepool. Nova seems to have no clear idea on instance count for the project displaying greater than 32k instances, and were also fighting DNS leaks all over the place making it unclear what is and is not an actual CI instance. I have a suspicion this DNS leak issue is related to the rate and tolerance of instance creation and lost messaging on the part of nodepool/rabbitmq. I talked to @thcipriani and @greg briefly post incident due to the difficult nature of debugging through this. I do not believe we can't go any further with nodepool without addressing these issues. I believe we are in agreement about this overall. The DNS leaks cleanup we are doing periodically is getting way out of hand and seems like a canary for deeper issues.

edit: yuvi made an issue for increasing head room here T142877