Page MenuHomePhabricator

CI mediawiki/core run times have increased since July 26th, 2019
Closed, ResolvedPublic

Description

Using the data collected by @dduvall in https://people.wikimedia.org/~dduvall/jenkins/ (using https://gerrit.wikimedia.org/g/releng/jenkins-build-stats/ ) I started graphing P-95 build-times for all projects (as an average), for mediawiki/* jobs (as an average), and for mediawiki/core (https://github.com/thcipriani/wikimedia-jenkins-stats/).

I noticed that there was an increase in the run time of jobs for mediawiki/core starting July 26th as seen in this graph:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 11 2019, 4:16 PM

Another reason: T231862: Selenium tests for Wikibase are being ran twice (Wikibase is part of gate, lots of selenium tests are being ran twice)

I think that matches with the deployment of quibble 0.0.33/0.0.34. Notably b2f9e363046696ff1f547a9724a705bbd52dbde2 Run node browser tests in each repo (was T199116).

I would guess T231862 is the main offender indeed and @awight has a patch for Quibble (Wikibase tests are run once because that is the project being run and second time because Wikibase is listed as a dependency of WikibaseLexeme).

Quibble 0.0.35 (with selenium de-duping) has been released and is being used since 2019-09-17, but the p95 hasn't dropped significantly (still around 20–22 minutes), and certainly not back to ~12 minutes.

Do we have an idea of what we can next do to improve this?

Latest graph, FYI:

Aklapper renamed this task from CI mediawiki/core run times have increased since July 26th to CI mediawiki/core run times have increased since July 26thm 2019.Sep 28 2019, 11:15 AM
Aklapper renamed this task from CI mediawiki/core run times have increased since July 26thm 2019 to CI mediawiki/core run times have increased since July 26th, 2019.

FWIW, these runtimes have since plunged mightily:

Krinkle closed this task as Resolved.Mar 26 2020, 3:28 PM
Krinkle claimed this task.
Krinkle added a subscriber: Krinkle.
Task description

From https://people.wikimedia.org/~dduvall/jenkins/2020-03-25/30-day/durations-by-project.html

Yep, we're back again around the 12-13 minute mark as was the status quo when this task was filed. I think that's all for this task? Boldly closing for now, but continuing with more ideas at T225730.

Krinkle reassigned this task from Krinkle to hashar.Mar 26 2020, 3:28 PM