Page MenuHomePhabricator

wmf-quibble-vendor-mysql-hhvm-docker job sometime take 40+ minutes to run
Closed, ResolvedPublic

Description

The gating job wmf-quibble-vendor-mysql-hhvm-docker sometimes takes more than 40 minutes to run which is annoying to say the less. That at least causes issue when doing SWAT which @Ladsgroup and @zeljkofilipin reported roughly 10 days ago.

I have put a slave offline since it was notably slow: https://integration.wikimedia.org/ci/computer/integration-slave-docker-1037/ Probably want to dig in all the wmf-quibble jobs and confirm whether that slave is routinely slow.

Event Timeline

hashar created this task.Apr 27 2019, 11:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 27 2019, 11:06 PM

On contint1001, I got the build time of all wmf-quibble jobs using:

xml_grep --text_only 'duration|builtOn' /srv/jenkins/builds/wmf-quibble*/*/build.xml

Then eventually once sorted the longest builds are all for 1037 for some reason.

Mentioned in SAL (#wikimedia-releng) [2019-04-30T01:35:39Z] <hashar> Deleting integration-slave-docker-1037 (bigram) it is too slow for some reason # T222023

hashar closed this task as Resolved.Apr 30 2019, 1:38 AM
hashar claimed this task.

And eventually a similar issue happens with the mediawiki core coverage job which I am tracking at T223971

Seems the reason for the slow down is due to the underlying CPU being slower than on other instances.