Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul
Open, NormalPublic

Description

We observed today a huge disruption of the distribution of jobs by Jenkins probably due to the Throttle plugin badly interacting with the Gearman plugin. For some reasons, the jobs in the queue seems to hold executors for a given node and that starves the number of executors.

When slaves will have only one executors, that will be essentially be solved since we will be able to remove the Throttle plugin.

A potential troublers is https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-publish/ mediawiki-core-doxygen-publish which is rather long (10+minutes) and triggers for mediawiki/core patch merges as well as tag. When we do a security release of MediaWiki core it is not unusual to have several patches triggers per release branches. So for REL1_26 and 3 changes A, B, C we end up triggering:

mediawiki-core-doxygen-publishA
mediawiki-core-doxygen-publishB
mediawiki-core-doxygen-publishC

Which takes half an hour, only to result in documentation for C which replaces doc generated for previous changes A and B.

Instead, we should just poll git from time to time and rebuild the doc for branches that have been updated or new tags that have been added. We will want to prevent the Jenkins git plugin from building legacy branches and tags and skip wmf branches/tags.

hashar created this task.Oct 16 2015, 8:40 PM
hashar updated the task description. (Show Details)
hashar raised the priority of this task from to Needs Triage.
hashar added subscribers: hashar, demon.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 16 2015, 8:40 PM
hashar triaged this task as Normal priority.Jan 5 2016, 12:29 PM
hashar set Security to None.

Change 347579 had a related patch set uploaded (by Hashar):
[integration/config@master] Decouple mw doxygen doc jobs for branches and tags

https://gerrit.wikimedia.org/r/347579

Change 347579 merged by Hashar:
[integration/config@master] Decouple mw doxygen doc jobs for branches and tags

https://gerrit.wikimedia.org/r/347579

Mentioned in SAL (#wikimedia-releng) [2017-04-12T08:52:47Z] <hashar> Cancelled bunch of mediawiki-core-doxygen-publish jobs that were keeping the queue busy/deadlocked builds. Should be moved to poll scm instead ( T115755 )