For example, the job for 797337 has been pending for almost 4 hours
(See also https://grafana-rw.wikimedia.org/d/000000322/zuul-gearman?orgId=1&from=now-6h&to=now&viewPanel=10, which suggests a more systemic issue)
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
The changes in the patch-performance or coverage have a low precedence and are only triggered after anything else.
The spike is probably what has caused the problem and it is slowly recovering.
Comment Actions
Ah, that makes sense, thank you @hashar — still, figuring out that spike would be useful 😄
Comment Actions
After a chat with @TheresNoTime @dancy and @Legoktm on IRC.
There has been a large spike between 15:13 TC and 15:33 UTC roughly. Most probably due to a series of patches send together to one of the big repositories such as mediawiki/core or Wikibase. That mostly has recovered by 15:49 UTC. This is a known issue of our outdated Zuul: T151089 T140297 . The Zuul merger is slow as well we would need more T222645 and do some optimization to git as well T307620 (there might be more tasks around). TLDR: our Zuul is obsolete.
After that until at least 22:00 UTC, there is a long standing queue of ~ 600 gearman functions waiting.
The reason found by @Legoktm is that LibUp kicked in, although it busy wait for test and gate-and-submit pipelines, it keeps them busy. Since the Zuul Gearman server primarily runs the low precedence pipeline if there is room, they end up never running. Additional the jobs stuck have a mutex in Zuul so only one of them can run at a time.
Thus, as LibUp keeps the higher precedence pipelines busy, there is a very little window for a job in the low precedence pipelines (coverage and patch-performance) to have a change to trigger their job. When they manage to get a job running, there is only one triggered due to the mutex.
We thus face a long tail which will resolve by itself.