For a few days now, Zuul has been lagging out to report completed builds back in Gerrit. There are most probably different root causes:
- When submitting a change, Zuul is locked, if Gerrit is slow to merge the whole process is locked down until the change is merged
- Zuul does not seem to recognize the LOST builds properly, specially if it is the last of a set of jobs. It seems to consider the change to be still around but does not bother reporting it since it is not FAIL nor SUCCESS
- Zuul did a ton of git remote update, I have reverted that patch an hour ago.
Usually Zuul become stuck between 8pm and 11pm GMT, which is the busy hours. European volunteers are very active, i18n bot is sending lot of patches and San Francisco is having a productive morning.
The signes of slowness are:
- https://integration.wikimedia.org/zuul/status has lot of changes with all build completed
- jenkins takes a long time to report back to gerrit even for very simple checks (such as the one on operations/puppet.git or translatewiki.net).
I have no idea what the fix is but upgrading Zuul is probably going to help. The new version of Zuul depends on a python module which is not available in Ubuntu Precise, I have packaged it and its pending review/merge/deploying (see bug 44061).