:) (note this is not a spam comment just that I like this)

Looking at debug messages over four days:

$ grep 'wmflabs.*running task' /var/log/nodepool/debug.log*|cut -d\  -f9|cut -d\. -f3|sort|uniq -c|sort -rn
   6266 ListServersTask
   3578 DeleteServerTask
   3578 CreateServerTask
   2267 ListFloatingIPsTask
     13 ListFlavorsTask
     13 ListExtensionsTask
      8 AddKeypairTask
      5 ListKeypairsTask
      5 GetServerTask
      5 DeleteKeypairTask
      2 FindImageTask

I looked at the patches this Friday morning and since I am out this evening with the week-end going, there was no chance for me to babysit the revert.

After discussion with @chasemp we will want to revert one by one making sure Chase is around to tweak the quota/rate and monitor the labs infra as needed. His Monday is quite busy, so we will do a first change on Tuesday and then I guess one per day.

Aim is to be done by end of next week?

Metrics:

https://grafana.wikimedia.org/dashboard/db/nodepool

https://grafana.wikimedia.org/dashboard/db/releng-zuul?panelId=25&fullscreen

https://grafana.wikimedia.org/dashboard/db/nodepool?panelId=6&fullscreen&from=1470866352522&to=1472162352523&var-provider=All&var-task=All

https://graphite.wikimedia.org/render/?width=966&height=489&_salt=1471549160.573&target=cactiStyle(zuul.pipeline.gate-and-submit.label.ci*.wait_time.mean)&hideLegend=false&lineMode=connected&from=-1d

@chasemp suggested to list out how many builds per timeframe a set of jobs does.

Looking at Zuul metrics, we get for each pipeline the amount of jobs triggered. I crafted a board that let us select multiple jobs and graph the aggregate:

https://grafana-admin.wikimedia.org/dashboard/db/zuul-job

Mentioned in SAL [2016-08-30T14:14:02Z] <hashar> Moved mediawiki-core-phpcs job back to Nodepool T143938

task cycle instructions

https://graphite.wikimedia.org/render/?width=1064&height=549&_salt=1472565876.337&target=cactiStyle(nodepool.task.wmflabs-eqiad.*.count)&areaMode=stacked&hideLegend=false&from=-12h

wait-time mean

https://graphite.wikimedia.org/render/?width=966&height=489&_salt=1471549160.573&target=cactiStyle(zuul.pipeline.gate-and-submit.label.ci*.wait_time.mean)&hideLegend=false&lineMode=connected&from=-6h

I don't entirely get the spike in trusty wait times with no corresponding trailing consumption of backlog. It seems fishy of zuul.

jessie job runtimes mean

https://graphite.wikimedia.org/render/?width=2064&height=1100&_salt=1472565940.394&areaMode=stacked&target=nodepool.job.*jessie*.runtime.mean&hideLegend=false&mode=stacked&from=-2h

trusty jobs runtime mean

https://graphite.wikimedia.org/render/?width=2064&height=1100&_salt=1472565940.394&target=nodepool.job.*trusty*.runtime.mean&hideLegend=false&from=-2h&lineMode=connected

jessie/trusty states

https://graphite.wikimedia.org/render/?width=1064&height=549&_salt=1472568888.402&target=cactiStyle(nodepool.target.gallium.wikimedia.org.ci-*-wikimedia.wmflabs-eqiad.*)&lineMode=connected&from=-3h&hideLegend=false

nodes at the ready

Change 307526 had a related patch set uploaded (by Rush):
nodepool: bump up ready states, max, and rate

https://gerrit.wikimedia.org/r/307526

notes on min-ready https://github.com/openstack-infra/nodepool/blob/master/nodepool/nodepool.py#L1389

Change 307526 merged by Rush:
nodepool: bump up ready states, max, and rate

https://gerrit.wikimedia.org/r/307526

Change 306725 had a related patch set uploaded (by Hashar):
Revert "Move tox-jessie & co. off of nodepool"

https://gerrit.wikimedia.org/r/306725

Change 308183 had a related patch set uploaded (by Hashar):
[labs/striker] port job to Nodepool Jessie instance

https://gerrit.wikimedia.org/r/308183

Change 308183 abandoned by Hashar:
[labs/striker] port job to Nodepool Jessie instance

Reason:
squashed in https://gerrit.wikimedia.org/r/#/c/306725/

https://gerrit.wikimedia.org/r/308183

Change 306725 merged by jenkins-bot:
Revert "Move tox-jessie & co. off of nodepool"

https://gerrit.wikimedia.org/r/306725

To bring the tox job back to permanent slaves:

• revert https://gerrit.wikimedia.org/r/#/c/306725/
• Rephrase the commit message one line summary since "Revert Revert Revert" tends to be confusing.
• Code-Review +2
• Wait for merge
• fab deploy_zuul from root of integration/config

I haven't deleted the Jenkins job, so the revert is essentially rolling back zuul/layout.yaml.

Before we move the composer jobs back, I'd like to implement T142457: Combine composer-php55 and composer-hhvm jobs.

The tox jobs are all happy.

The rake jobs have been made to only trigger when ruby related files are modified (T144325). It is done to only a few builds per days so I will process tomorrow:

Then prepare https://gerrit.wikimedia.org/r/#/c/306722/ "Revert "Move npm-node-4 off of nodepool""" which has more jobs to add to Nodepool.

Mentioned in SAL [2016-09-07T09:41:22Z] <hashar> Moving rake jobs back to Nodepool ( T143938 ) with https://gerrit.wikimedia.org/r/#/c/306723/ and https://gerrit.wikimedia.org/r/#/c/306724/

I have moved the rake and oojs-ui-rake jobs to Nodepool. Validated them by hitting recheck on a couple dummy changes:

Change 306722 had a related patch set uploaded (by Hashar):
Revert "Move npm-node-4 off of nodepool"

https://gerrit.wikimedia.org/r/306722

Change 306722 abandoned by Hashar:
Revert "Move npm-node-4 off of nodepool"

Reason:
That patch is a non sense. The commit it reverts only switched 'npm-node-4' job, leaving all the others to Nodepool :D

https://gerrit.wikimedia.org/r/306722

Change 313061 had a related patch set uploaded (by Hashar):
Bring back npm-node-4 to Nodepool

https://gerrit.wikimedia.org/r/313061

Change 313061 merged by jenkins-bot:
Bring back npm-node-4 to Nodepool

https://gerrit.wikimedia.org/r/313061

Mentioned in SAL (#wikimedia-releng) [2016-09-27T20:11:46Z] <hashar> Reloading Zuul to deploy 3c3289aa1a for T143938 and T146783

I have moved the node-npm-4 job back. Is roughly a dozen of builds per hours, hardly a dent.

From a discussion with Chase. What is left todo is to migrate back the PHP based jobs:

https://gerrit.wikimedia.org/r/#/c/306727/
Revert "Temporarily move composer-hhvm/php5 jobs off of nodepool"

That patch would probably move to many build at once so it is better to split it up in manageable chunk. This week is not ideal to accomplish the move, so better done starting next monday.

Change 306727 had a related patch set uploaded (by Hashar):
Revert "Temporarily move composer-hhvm/php5 jobs off of nodepool"

https://gerrit.wikimedia.org/r/306727

I have rebased the patch https://gerrit.wikimedia.org/r/#/c/306727/ Revert "Temporarily move composer-hhvm/php5 jobs off of nodepool"

That replace three jobs that runs on permanent slaves: composer-hhvm, composer-php55 and composer-php70.

Expected build load shifted to Nodepool instances is roughly 350 builds per day which would be added to the 800/1000 builds per day we are already doing on Nodepool. That bring us back to the situation we had from June to early August 2016.

Details

https://grafana.wikimedia.org/dashboard/db/zuul-job?var-pipeline=All&var-job=%7Bcomposer-hhvm,composer-php55,composer-php70%7D&var-status=All&panelId=5&fullscreen . View over a month:

There is currently 800 - 1000 builds per day on Nodepool instances: https://grafana-admin.wikimedia.org/dashboard/db/nodepool?panelId=23&fullscreen

Change 314278 had a related patch set uploaded (by Hashar):
Recreate jobs for composer-hhvm/php on Nodepool

https://gerrit.wikimedia.org/r/314278

Change 314278 merged by jenkins-bot:
Recreate jobs for composer-hhvm/php on Nodepool

https://gerrit.wikimedia.org/r/314278

Change 306727 merged by jenkins-bot:
Move composer-hhvm/php5 jobs back to Nodepool

https://gerrit.wikimedia.org/r/306727

Mentioned in SAL (#wikimedia-releng) [2016-10-13T20:12:26Z] <hashar> Switching composer-hhvm / composer-php55 to Nodepool https://gerrit.wikimedia.org/r/#/c/306727/ T143938