Page MenuHomePhabricator

Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul
Closed, ResolvedPublic


We observed today a huge disruption of the distribution of jobs by Jenkins probably due to the Throttle plugin badly interacting with the Gearman plugin. For some reasons, the jobs in the queue seems to hold executors for a given node and that starves the number of executors.

When slaves will have only one executors, that will be essentially be solved since we will be able to remove the Throttle plugin.

A potential troublers is mediawiki-core-doxygen-publish which is rather long (10+minutes) and triggers for mediawiki/core patch merges as well as tag. When we do a security release of MediaWiki core it is not unusual to have several patches triggers per release branches. So for REL1_26 and 3 changes A, B, C we end up triggering:


Which takes half an hour, only to result in documentation for C which replaces doc generated for previous changes A and B.

Instead, we should just poll git from time to time and rebuild the doc for branches that have been updated or new tags that have been added. We will want to prevent the Jenkins git plugin from building legacy branches and tags and skip wmf branches/tags.

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added subscribers: hashar, demon.
hashar triaged this task as Medium priority.Jan 5 2016, 12:29 PM
hashar set Security to None.

Change 347579 had a related patch set uploaded (by Hashar):
[integration/config@master] Decouple mw doxygen doc jobs for branches and tags

Change 347579 merged by Hashar:
[integration/config@master] Decouple mw doxygen doc jobs for branches and tags

Mentioned in SAL (#wikimedia-releng) [2017-04-12T08:52:47Z] <hashar> Cancelled bunch of mediawiki-core-doxygen-publish jobs that were keeping the queue busy/deadlocked builds. Should be moved to poll scm instead ( T115755 )

Change 415588 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] WIP Polling job for MediaWiki doxygen

Some got magically generated via

Gotta send an email on failure and JJB fails to honor the git reference repository parameter :( is now polling the SCM. It is not using a git reference repository, but it does not really matter.

Change 415588 merged by jenkins-bot:
[integration/config@master] Polling job for MediaWiki doxygen