The gating job wmf-quibble-vendor-mysql-hhvm-docker sometimes takes more than 40 minutes to run which is annoying to say the less. That at least causes issue when doing SWAT which @Ladsgroup and @zeljkofilipin reported roughly 10 days ago.
I have put a slave offline since it was notably slow: https://integration.wikimedia.org/ci/computer/integration-slave-docker-1037/ Probably want to dig in all the wmf-quibble jobs and confirm whether that slave is routinely slow.
On contint1001, I got the build time of all wmf-quibble jobs using:
xml_grep --text_only 'duration|builtOn' /srv/jenkins/builds/wmf-quibble*/*/build.xml
Then eventually once sorted the longest builds are all for 1037 for some reason.
Mentioned in SAL (#wikimedia-releng) [2019-04-30T01:35:39Z] <hashar> Deleting integration-slave-docker-1037 (bigram) it is too slow for some reason # T222023
And eventually a similar issue happens with the mediawiki core coverage job which I am tracking at T223971
Seems the reason for the slow down is due to the underlying CPU being slower than on other instances.