The job mediawiki-core-code-coverage-docker usually takes 2h / 2h30 based on https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/buildTimeTrend
integration-slave-docker-1054 is usually less than two hours. The job times out after four hours on 1055 and 1056 :(
After some investigations below, the slow instances are on cloudvirt1005 / cloudvirt1006 which have Intel Xeon E3-12xx v2 (Ivy Bridge) at 2.7Ghz. The fasted builds are on instances having a Skylake processor at 2.3Ghz.
I can imagine a newer architecture offers improvement, but a for a single thread CPU bound doing simple maths, I would expect the Ivy Bridge Xeon at 2.7Ghz to be faster than the Skylake one at 2.3Ghz.
I tried a couple small CPU benchmarks which I ran on instances:
- Compute 10k prime numbers using 64 bit with the sysbench package and running: sysbench --test=cpu run.
- Do some basic math with stress-ng and running: stress-ng --cpu 1 --cpu-ops=400000
|Model||Intel Xeon E3-12xx v2 (Ivy Bridge)||Intel Core Processor (Skylake)|
|cpu MHz||2,7 Ghz||2,3 GHz|
|sysbench duration||~ 16 seconds||~ 9 seconds|
|stress-ng||~ 18 seconds||~ 10 seconds|
Rearranged, with max turbo:
|Host||Model||base speed||Max turbo||bogoMips||Sysbench|
|cloudvirt1006||Xeon E5-2697 v2||2,700 MHz||3,500 MHz||5,386||15.13s|
|cloudvirt1025||Xeon Gold 6140||2,300 MHz||3,700 MHz||4,590||9.24s|
|cobalt||Xeon E5-2623 v3||3,000 MHz||3,500 MHz||6,000||9,30s|
|contint1001||Xeon E5-2640 v3||2,600 MHz||3,400 MHz||5,200||9.37s|
|@hashar||i7-8550U||1,800 MHz||4,000 MHz||4,000||7,43s|
|@hashar #2||i5-4250U||1,300 MHz||2,600 MHz||3,800||11,8s|
Note how despite the bogoMips and CPU speed being higher on the Xeon Ivy Bridge, it performs twice slower.
I really dont get why the Intel Xeon is so slow :-/ Maybe it is an oddity due to kvm or a BIOS / hardware configuration issue. One would have to run the same benchmarks on the real servers for comparison?