Change Details

The job `mediawiki-core-code-coverage-docker` usually takes 2h / 2h30 based on https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/buildTimeTrend integration-slave-docker-1054 is usually less than two hours. The job times out after four hours on 1055 and 1056 :( After some investigations below, the slow instances are on cloudvirt1005 / cloudvirt1006 which have `Intel Xeon E3-12xx v2 (Ivy Bridge)` at 2.7Ghz. The fasted builds are on instances having a Skylake processor at 2.3Ghz. I can imagine a newer architecture offers improvement, but a for a single thread CPU bound doing simple maths, I would expect the Ivy Bridge Xeon at 2.7Ghz to be faster than the Skylake one at 2.3Ghz. I tried a couple small CPU benchmarks which I ran on instances: * Compute 10k prime numbers using 64 bit with the `sysbench` package and running: `sysbench --test=cpu run`. * Do some basic math with `stress-ng` and running: `stress-ng --cpu 1 --cpu-ops=400000` Results: | Model | Intel Xeon E3-12xx v2 (Ivy Bridge) | Intel Core Processor (Skylake) |--|--|-- | cpu MHz | 2,7 Ghz | 2,3 GHz | bogoMips | 5390 | 4590 | sysbench duration | ~ 16 seconds | ~ 9 seconds | stress-ng | ~ 18 seconds | ~ 10 seconds Rearranged, not taking in account the turbo speed: | Host | Model | base speed | bogoMips | Sysbench |---------------|-----------------|------------|----------|--------- | cloudvirt1006 | Xeon E5-2697 v2 | 2,700 MHz | 5,386 | **15.13s** | cloudvirt1025 | Xeon Gold 6140 | 2,300 MHz | 4,590 | **9.24s** | cobalt | Xeon E5-2623 v3 | 3,000 MHz | 6,000 | **9,30s** | contint1001 | Xeon E5-2640 v3 | 2,600 MHz | 5,200 | **9.37s** | @hashar | i7-8550U | 1,800 MHz | 4,000 | **7,43s** Note how despite the bogoMips and CPU speed being higher on the Xeon Ivy Bridge, it performs twice slower. I really dont get why the Intel Xeon is so slow :-/ Maybe it is an oddity due to kvm or a BIOS / hardware configuration issue. One would have to run the same benchmarks on the real servers for comparison?

The job `mediawiki-core-code-coverage-docker` usually takes 2h / 2h30 based on https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/buildTimeTrend integration-slave-docker-1054 is usually less than two hours. The job times out after four hours on 1055 and 1056 :( After some investigations below, the slow instances are on cloudvirt1005 / cloudvirt1006 which have `Intel Xeon E3-12xx v2 (Ivy Bridge)` at 2.7Ghz. The fasted builds are on instances having a Skylake processor at 2.3Ghz. I can imagine a newer architecture offers improvement, but a for a single thread CPU bound doing simple maths, I would expect the Ivy Bridge Xeon at 2.7Ghz to be faster than the Skylake one at 2.3Ghz. I tried a couple small CPU benchmarks which I ran on instances: * Compute 10k prime numbers using 64 bit with the `sysbench` package and running: `sysbench --test=cpu run`. * Do some basic math with `stress-ng` and running: `stress-ng --cpu 1 --cpu-ops=400000` Results: | Model | Intel Xeon E3-12xx v2 (Ivy Bridge) | Intel Core Processor (Skylake) |--|--|-- | cpu MHz | 2,7 Ghz | 2,3 GHz | bogoMips | 5390 | 4590 | sysbench duration | ~ 16 seconds | ~ 9 seconds | stress-ng | ~ 18 seconds | ~ 10 seconds Rearranged, with max turbo: | Host | Model | base speed | Max turbo | bogoMips | Sysbench |---------------|-----------------|------------|-----------|--------- | cloudvirt1006 | Xeon E5-2697 v2 | 2,700 MHz | 3,500 MHz | 5,386 | **15.13s** | cloudvirt1025 | Xeon Gold 6140 | 2,300 MHz | 3,700 MHz | 4,590 | **9.24s** | cobalt | Xeon E5-2623 v3 | 3,000 MHz | 3,500 MHz | 6,000 | **9,30s** | contint1001 | Xeon E5-2640 v3 | 2,600 MHz | 3,400 MHz | 5,200 | **9.37s** | @hashar | i7-8550U | 1,800 MHz | 4,000 MHz | 4,000 | **7,43s** | @hashar #2 | i5-4250U | 1,300 MHz | 2,600 MHz | 3,800 | **11,8s** Note how despite the bogoMips and CPU speed being higher on the Xeon Ivy Bridge, it performs twice slower. I really dont get why the Intel Xeon is so slow :-/ Maybe it is an oddity due to kvm or a BIOS / hardware configuration issue. One would have to run the same benchmarks on the real servers for comparison?

The job `mediawiki-core-code-coverage-docker` usually takes 2h / 2h30 based on https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/buildTimeTrend integration-slave-docker-1054 is usually less than two hours. The job times out after four hours on 1055 and 1056 :( After some investigations below, the slow instances are on cloudvirt1005 / cloudvirt1006 which have `Intel Xeon E3-12xx v2 (Ivy Bridge)` at 2.7Ghz. The fasted builds are on instances having a Skylake processor at 2.3Ghz. I can imagine a newer architecture offers improvement, but a for a single thread CPU bound doing simple maths, I would expect the Ivy Bridge Xeon at 2.7Ghz to be faster than the Skylake one at 2.3Ghz. I tried a couple small CPU benchmarks which I ran on instances: * Compute 10k prime numbers using 64 bit with the `sysbench` package and running: `sysbench --test=cpu run`. * Do some basic math with `stress-ng` and running: `stress-ng --cpu 1 --cpu-ops=400000` Results: | Model | Intel Xeon E3-12xx v2 (Ivy Bridge) | Intel Core Processor (Skylake) |--|--|-- | cpu MHz | 2,7 Ghz | 2,3 GHz | bogoMips | 5390 | 4590 | sysbench duration | ~ 16 seconds | ~ 9 seconds | stress-ng | ~ 18 seconds | ~ 10 seconds Rearranged, not taking in account thewith max turbo speed:: | Host | Model | base speed | Max turbo | bogoMips | Sysbench |---------------|-----------------|------------|-----------|--------- | cloudvirt1006 | Xeon E5-2697 v2 | 2,700 MHz | 3,500 MHz | 5,386 | **15.13s** | cloudvirt1025 | Xeon Gold 6140 | 2,300 MHz | 3,700 MHz | 4,590 | **9.24s** | cobalt | Xeon E5-2623 v3 | 3,000 MHz | 3,500 MHz | 6,000 | **9,30s** | contint1001 | Xeon E5-2640 v3 | 2,600 MHz | 3,400 MHz | 5,200 | **9.37s** | @hashar | i7-8550U | 1,800 MHz | 4,000 MHz | 4,000 | **7,43s** | @hashar #2 | i5-4250U | 1,300 MHz | 2,600 MHz | 3,800 | **11,8s** Note how despite the bogoMips and CPU speed being higher on the Xeon Ivy Bridge, it performs twice slower. I really dont get why the Intel Xeon is so slow :-/ Maybe it is an oddity due to kvm or a BIOS / hardware configuration issue. One would have to run the same benchmarks on the real servers for comparison?