For context, the actual time to run the tests for operations/puppet is under one minute for most patches.
Either Zuul or jenkins are broken, and this has been a constant pain in the last few weeks for everyone involved.
Triaging to UBN! because this is effectively an outage of the service.
The root cause was a faulty patch merged in mediawiki/core on Friday. It roughly doubled the time it takes to run the tests so that eg Wikibase changes occupied execution slot for up to an hour. In turn that starved the very thin pool of executors we currently have which thus delayed execution of jobs for non MediaWiki repo.
Anyway, that has been fixed by reverting the faulty code.