Our migration from grunt-contrib-qunit (using PhantomJS, hardcoded task in Jenkins job) to Karma (using Chrome/Firefox and the local npm-test entry point) is going well for standalone JavaScript projects (OOjs, VisualEditor).

These have since been promoted to be part of the main test pipeline and replaced their PhantomJS counterparts as this was blocking feature development and tests using newer browser features (and it significantly sped up the test due to Chromium 38+ being much faster than PhantomJS 1.9.x)

However for MediaWiki core and extensions, the Karma job (currently non-voting, work in progress) is failing most of the time due to the Chrome instance disconnecting half-way through the running of the qunit tests.

So far it seems to be consistently timing out at test 142/248. This is the mediawiki.jqueryMsg.test that makes various HTTP requests to load.php. The logic behind it seems solid (works locally for developers running tests in Chrome/Firefox, and is still working fine in Jenkins using PhantomJS today).

I initially suspected the way Apache is serving files through MediaWiki PHP (as opposed to the Node.js static file server that the PhantomJS test used) may be the culprit, but then the PhantomJS server also defers to Apache and MediaWiki for most files already.

I suspect it's caused by the Karma job running on slaves in labs (which are inherently slower than the production slaves PhantomJS runs on). This slowdown may be causing it to hit a timeout somewhere. Likely related to MediaWiki using I/O-bound cache files and the SQLite database.