This morning the Zuul coverage pipeline is filled up with changes all waiting for the job mwext-phpunit-coverage-patch to eventually run.
I enabled a mutex on that job to have only one build of it running at any given time https://gerrit.wikimedia.org/r/#/c/419674/
The CirrusSearch change https://gerrit.wikimedia.org/r/#/c/419804/ had a PS2 trigger the coverage job, immediately followed by a PS3. The job for PS2 got canceled by Zuul as intended, but the mutex did not get released. Hence the mutex is hold and no build of that job can run anymore.
The reason is our Zuul version does not release the mutex on build cancellation. Luckily Tobias Henkel has backported a fix to have zuul release the mutex on job cancellation:
Zuul v2 https://review.openstack.org/#/c/384980/
Zuul v3 https://review.openstack.org/#/c/432211/