This sounds good to me. We can go forward with it if @hashar has no complaints. 😄

When we add an extension to the shared gate job, we automatically pick up its QUnit and PHPUnit tests, but the selenium tests need to be registered manually in mediawiki/core.

When npm run selenium runs, wdio.conf.js automatically finds tests in all extensions and skins:

 specs: [
   relPath( './extensions/*/tests/selenium/specs/**/*.js' ),
   relPath( './skins/*/tests/selenium/specs/**/*.js' )
],

so there is no need to add Selenium tests manually to the config file, in development environment or CI. Or, did I misunderstood you? 😄

@zeljkofilipin Yeah, misunderstanding. But I realise I did forget to address the specs discovery part and how it relates.

Summary

In summary, I expect two problems:

1. The selenium test do not need to be run from core because their run-time does not depend on core.

2. The JavaScript code containing the Selenium tests are standalone, and must not be included directly from code in core.

The Selenium tests are more like Codesniffer and ESLint than like PHPUnit. The reason being that the Selenium tests do not share their execution environment (Node.js) with any code from MediaWiki core (browser JS, PHP).

For PHPUnit tests, we need to run the tests from core for two reasons:

• The PHPUnit process needs to discover test suites. MediaWiki has a custom PHPUnit test-registration system (using MediaWiki hooks) which means PHPUnit on its own cannot discover all the test suites. The logic for this is in various hooks and other code in core.git.
• The PHPUnit process needs to include tests, and these tests need to include extension classes, which in turn can access or extend classes from MediaWiki core. This is all dependant on the autoloader which is part of MediaWiki core.

The fact that we have to run PHPUnit from core is in my opinion technical debt and imposes many limitations and constraints that affect us from time to time. But... at least we can see that it is not an easy problem to solve, and due to the above it makes sense why we currently have to run PHPUnit tests from core - with the PHPUnit version only specified in core's composer.json.

For the Selenium tests, none of these restrictions apply. There is no MediaWiki core code executed in the Node.js process that drives Selenium tests. And there is no special logic in MediaWiki core required to discover Selenium test suites. The test suites are passed directly to the wdio process, which includes them, and those test suites only interact with Node packages available from npm. There are no other implied dependencies or constraints, except that there is an environment variable pointing to an installation of MediaWiki reachable over HTTP.

As demonstrated with the ORES and Echo repos, the tests are already runnable from a local directory outside of core. However, while this fine locally, it only works by accident in CI. CI currently still runs Selenium tests for all installed extensions from core. This is a problem:

• There is now an implicit requirement that an extensions' own Selenium test suites cannot have extra development dependencies besides core's.
• The development dependencies it has must match the versions in core.
• This is a maintainability problem, and will make it very difficult to upgrade.
• This will make it difficult to scale to more extensions, including extensions not deployed by Wikimedia Foundation.

We have previously made this mistake a few times before, and each time we learned to not do it after we found ourselves with problems. For example:

JSHint: Lesson learned

In 2012, we installed JSHint globally on Jenkins slaves. And all our Gerrit repositories could opt-in to enable a JSHint job. This worked great for a while, but we quickly found a problem: Running tests locally did not produce the same errors because users did not have the same version of JSHint. They had to keep their package.json in sync with Jenkins config. (This is analogous to how the standalone Selenium test in extensions now has to match core, because CI ignores the local package.json.)

An even bigger problem arose when we tried to upgrade JSHint to a newer version. We could not do it because changing the version installed centrally, immediately breaks all hundreds repositories. We cannot magically change all repositories atomically at once. And even if we could, it would require changes to many many repositories and force lots of teams to do unscheduled maintenance for something that is not important and should be a decision made by individual teams when they want to do it.

We solved this problem by changing to npm test jobs, where individual repositories decide which version of JSHint is used, and which rule configuration. This does not mean everybody should use different version and different rules. But it us to slowly evolve and get better without being disruptive.

The lesson learned is that by decentralising the JSHint version, we were able to upgrade quicker and be more consistent than if it were centralised.

This system also allowed us to slowly switch from JSHint to ESLint. And all of this required zero cost from release engineering, because the Jenkins job is only responsible for running npm test.

Conclusion for Selenium

So I propose the following:

• Update the Jenkins job for selenium to run npm install && npm run selenium-test from each installed extension repository.
• Update mediawiki/core spec to not include extensions/*/tests. This also fixes the problem some people have locally where it is impossible to disable an extension by commenting out in LocalSettings, because selenium still hard-includes it, making the test fail.

This means:

• The version of wdio, Mocha etc, can be controlled per-repository.
• You can try improvements in one repository without atomically breaking other extensions. Discover any issues and address them as we go. Think about what if we want to change from Mocha to something else later? E.g. JSHint-ESLint migration, Codesniffer 2:3 migration, etc.
• Libraries like mwbot or cucumber can be added by repos that need it, without core needing to have every dev dependency of every extension, which would make core.git a bottleneck at first, and later become impossible to upgrade.

In T193943#4313350, @Krinkle wrote:

1. The selenium test do not need to be run from core because their run-time does not depend on core.

Already the case for T188742. Tests target beta cluster and the dependencies are only node, npm and the extension/skin repository. Core is not needed.

In T193943#4313350, @Krinkle wrote:

• Update the Jenkins job for selenium to run npm install && npm run selenium-test from each installed extension repository.

Thanks @Krinkle for a detailed explanation. I have not strong opinions and you obviously have more experience with this. As far as I see, this should be implemented in Quibble. I already plan to work on making npm install && npm run selenium-test work for extensions for T188742.

• Update mediawiki/core spec to not include extensions/*/tests. This also fixes the problem some people have locally where it is impossible to disable an extension by commenting out in LocalSettings, because selenium still hard-includes it, making the test fail.

This should be trivial to do once CI is updated.