Fatal errors, like calling a non-existing method, are not visible in Jenkins output. Not in the console output, not in the build artifacts, and not under "test results" either. So, if e.g. an extension becomes incompatible with core in a way that triggers a fatal error, Jenkins will start failing for that extension, with no indication as to why.
For reference, see the test failure reported on PS2 of https://gerrit.wikimedia.org/r/#/c/194516/
It appears to be limited to the mwext-Wikibase-client-tests job, the repo ones are fine.
@Legoktm so maybe it's an issue with the php output buffer? I would expect php to discard any buffers before reporting a fatal, but who knows.
Perhaps the best approach would be not write errors to a log file directly, and not to stdout/stderr at all, to sidestep any issues with buffering. What is error_log set to?
Hmm, is something in client messing with the output buffer? I grepped for "ob_" but didn't see anything client-related.
client tests have Scribunto and cldr as dependencies, repo just has cldr.
jenkins just passes client to Wikibase/build/jenkins/mw-apply-wb-settings.sh, so that's where the other differences are.
Is this still an issue? This might be related to T85799 / 0a35ec5d1f370cee4ebb9be0c386978d6cfcaabb.