Split from T72597#5518854
A related issues are the mwext-codehealth jobs, they are made to not run concurrently. Sometime one would see several of them pending in the Jenkins build queue and some of the Jenkins agents are idling although they should be running jobs.
The builds are queued by Jenkins, but the Gearman plugin already assigned a node for those builds. The node assignment can be seen in /var/lib/jenkins/queue.xml:
<hudson.model.Queue_-State> <items> <hudson.model.Queue_-BlockedItem> <actions> <hudson.plugins.gearman.NodeAssignmentAction plugin="gearman-plugin@0.2.0.3.e27817f"> <labelAtom>integration-agent-docker-1009</labelAtom> </hudson.plugins.gearman.NodeAssignmentAction> ... <hudson.model.Queue_-BlockedItem> <actions> <hudson.plugins.gearman.NodeAssignmentAction plugin="gearman-plugin@0.2.0.3.e27817f"> <labelAtom>integration-agent-docker-1005</labelAtom> </hudson.plugins.gearman.NodeAssignmentAction>
I don't have the details, but Gearman is thus unable to use any of the executors on those two nodes until the build queued by Jenkins starts executing.
It might be related to the lock we occasionally have for deployment-prep.