Page MenuHomePhabricator

wmf-quibble-vendor-mysql-hhvm-docker job sometime take 40+ minutes to run
Closed, ResolvedPublic


The gating job wmf-quibble-vendor-mysql-hhvm-docker sometimes takes more than 40 minutes to run which is annoying to say the less. That at least causes issue when doing SWAT which @Ladsgroup and @zeljkofilipin reported roughly 10 days ago.

I have put a slave offline since it was notably slow: Probably want to dig in all the wmf-quibble jobs and confirm whether that slave is routinely slow.

Event Timeline

On contint1001, I got the build time of all wmf-quibble jobs using:

xml_grep --text_only 'duration|builtOn' /srv/jenkins/builds/wmf-quibble*/*/build.xml

Then eventually once sorted the longest builds are all for 1037 for some reason.

Mentioned in SAL (#wikimedia-releng) [2019-04-30T01:35:39Z] <hashar> Deleting integration-slave-docker-1037 (bigram) it is too slow for some reason # T222023

hashar claimed this task.

And eventually a similar issue happens with the mediawiki core coverage job which I am tracking at T223971

Seems the reason for the slow down is due to the underlying CPU being slower than on other instances.