From a conversation with Zeljko related to webdriver.io testing

Given a patch to the the mediawiki extension RelatedArticles. Zuul injects a parameter EXT_NAME=RelatedArticles

Zuul then adds direct extensions dependencies: ['BetaFeatures', 'Cards', 'MobileFrontend'] which each bring more to it (eg: MobileFrontend also depends on ['Echo', 'VisualEditor', 'ZeroBanner', 'MobileApp']

The Webdriver.io jenkins job then:

cd MW_INSTALL_PATH
npm run selenium

That ends up running every available tests because the wdio config has a glob pattern extensions/*/tests/selenium/**/*.js.

We would need to use subsets for example:

Run tests solely for the extension that sent the patch

wdio --spec extensions/${EXT_NAME}/tests/selenium/**/*.js

MediaWiki sub set of tests:

wdio --spec tests/selenium/smoke/**/*.js

Smokes from other extensions

wdio --spec extensions/${EXT_NAME}/tests/selenium/smoke/**/*.js

That came up also on a patch for Echo https://gerrit.wikimedia.org/r/#/c/313221/ which added a test requiring Flow behavior. I commented about how we could inject Flow as a dependency but that means all Flow tests will be run as well.

The same split has to be done for PHPUnit as well

Yurik and Hashar had a discussion on Dec. 14th 2016 about JSonConfig also running all the Scribunto tests, which is a recurring issue affecting multiple extensions (eg: T126670).

IRC logs starting at 09:00 https://wm-bot.wmflabs.org/logs/%23wikimedia-releng/20161214.txt

A lame drawing showing how the PHPUnit tests could be split: https://docs.google.com/drawings/d/1ebVAFqM80Me7HCiLV-JdAkNCh3unLCGsuNqRcgiGtEI