I've seen this for several years, and each time I convinced myself it's a temporary issue due to ... something something. I also suspect that for a while in 2016-2017, this wasn't an issue. But maybe I just didn't notice it.
Let me report it here in case it's a known issue and/or is something we can fix.
Quite often I see that the Zuul status page has dozens of stalled jobs (queued) with only one or two things actually being executed. Meanwhile Jenkins shows dozens of idle executor slots are available, seemingly not doing anything.
Below isn't the worst example of this, as it still shows 5 change sets being processed (16 jobs in execution phase). But, it still demonstrates the issue of dozens of jobs being stalled while executor slots appear to be available!
Zuul status page
- executing: 16 jobs
- waiting: 41 jobs (queued)
Jenkins executor overview (docker hosts)
- in use: 17 executors (including 1 non-zuul job for selenium, from a cron timer)
- idle: 55 executors