There is a flaky test in the Scribunto extension that started to repeatedly fail for the same reason, wrecking my nerves and doing nothing good but (literally) wasting time:
Just one of many recent examples: https://integration.wikimedia.org/ci/job/mwext-testextension-php55-composer/4322/console
16:09:00 1) LuaSandbox: Scribunto_LuaSandboxTests::testArgumentParsingTime 16:09:00 Unused arguments not counted in preprocess 16:09:00 Failed asserting that 0.55842105400000008 is less than 0.25. 16:09:00 16:09:00 /mnt/jenkins-workspace/workspace/mwext-testextension-php55-composer/src/extensions/Scribunto/tests/engines/LuaSandbox/SandboxTest.php:75
The trivial reason why this fails is that this runs on a virtualized CI machine. There are other things running the same time, delaying execution. One simply can not rely on any minimum or maximum execution time of a specific snippet of code. This is guaranteed to fail.
Suggested solutions:
- https://gerrit.wikimedia.org/r/305499 removes any code that measures CPU cycles with trivial counts. In consequence the test does not test any more what it was intended to test. The counts the patch introduces are correct: The hook is expected to be called three times. But the first two times should not add on top of the execution timer and the result getCPUUsage returns (search for luasandbox_timer_pause and luasandbox_timer_unpause in the luasandbox extension). @Anomie decided to block this patch with no way forward.
- A proper solution would be to mock the code that collects the benchmark results this test is currently testing. But the code doing this is not even part of this extension! Search for getCPUUsage. This is in luasandbox, which is a PHP extension written in C. I believe it's not even possible to mock this.
- https://gerrit.wikimedia.org/r/305620 just kills the thing.
- Remove Scribunto from the Wikibase CI. Seriously. If the extension is broken and we can not rely on their tests any more, having them in our CI is of no value any more. The contrary: all it does at the moment is causing harm. We can re-add it later when the issue is solved.