| Paladox | |
| Jan 16 2017, 7:07 PM |
| F5294297: openstack_api_throttling.png | |
| Jan 16 2017, 8:45 PM |
| F5294299: pool.png | |
| Jan 16 2017, 8:45 PM |
| F5294295: zuul_changes.png | |
| Jan 16 2017, 8:45 PM |
See https://integration.wikimedia.org/zuul/
Some of the repo's show they have been waiting ~40mins.
There is indeed 20-40 changes in Zuul pipelines since 17:30 UTC
Nodepool queries to OpenStack are rate limited at one every six seconds and we have reached that limit for the last 3 hours:
The pool of instance is rather busy as well:
Looks like lot of changes have been sent.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T155444 (Nodepool) CI is really slow tonight | |||
| Resolved | • Prtksxna | T155483 Speed up oojs/ui Jenkins jobs |
One other possibility is we had a lot of changes merged for oojs/ui. On merge that runs oojs-ui-coverage but only one copy of the job can run on the infra. However for each job pending, it seems Jenkins allocate a node which thus is sitting idle/cant process another job. That is just a theory really.
I've been submitting a lot of OOjs UI changes today, and James has been merging a lot of them. Sorry if we overwhelmed the CI. :D
(There were 24 changesets submitted in the last 3 hours, some with multiple patchsets. https://gerrit.wikimedia.org/r/#/q/project:oojs/ui)
I guess that explains it oojs/ui is an heavy consumer with long running jobs. Taking https://gerrit.wikimedia.org/r/#/c/332344/ as an example given one propose a patch, +2 it and then jobs run after merge we got:
| test | time |
|---|---|
| composer-package-hhvm-trusty | 44s |
| composer-package-php55-trusty | 48s |
| oojs-ui-npm-node-4-jessie | 12m 05s |
| oojs-ui-npm-run-demos-node-4-jessie | 9m 30s |
| oojs-ui-npm-run-doc-node-4-jessie | 8m 35s |
| gate | time |
| composer-package-hhvm-trusty | 41s |
| composer-package-php55-trusty | 37s |
| oojs-ui-npm-node-4-jessie | 10m 16s |
| oojs-ui-npm-run-demos-node-4-jessie | 9m 02s |
| oojs-ui-npm-run-doc-node-4-jessie | 5m 55s |
| post merge | time |
| oojs-ui-jsduck-publish | 9m 31s |
| oojs-ui-doxygen-publish | 9s |
| oojs-ui-coverage | 12m 06s |
| oojs-ui-demos-publish | 11m 24s |
So if one send a patch and +2 it that is six Jessie instances being busy for 6 minutes or four during 9-10 minutes. It only takes three such patches to consume the whole pool.