Change 446823 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/extensions/Popups@master] QA: Selenium no more needs wgUsejQueryThree

https://gerrit.wikimedia.org/r/446823

Change 446823 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] QA: Selenium no more needs wgUsejQueryThree

https://gerrit.wikimedia.org/r/446823

VisualEditor	$wgVisualEditorShowBetaWelcome = false;	Disable the welcome menu which breaks MediaWiki core Selenium tests

If I recall correctly, the VisualEditor welcome dialog can also be hidden by others means (cookie, user preference, and url parameter). I'm guessing the problem is not with VisualEditor's own tests (which indeed, could use cookies or url parameters), but the problem is with MediaWiki core's selenium tests not passing when VE is installed?

That raises an important question: Do we want to test features in isolation, or test (shared gate extension) features together?

Isolation

If we test features in isolation, this means core tests run without extensions installed, and extension tests run with only that extension (+dependencies) installed.

Testing in isolation, means we can test any core feature, even those replaced or modified by an extension. Extensions that replace the "Search" or "Create account" page can be tested, and also the core default version.

Together

If we test the result of installing everything together, then some core features cannot be tested, and also, extensions cannot test all certain features.

• An extension that modifies or adds to a core feature, such as the welcome dialog from VisualEditor, do we disable that so that the core test passes? What if VisualEditor wants to test that the welcome dialog works? If we disable it, that also means the test no longer represents the normal configuration of the extension.
• A change that breaks a core feature, but its breakage is repaired/masked by an extension – how do we catch this regression from MediaWiki core's tests?

Currently

We currently test everything together. But, at the same time, we use selenium/LocalSettings.php to "disable" interference from extensions with core features. This sounds like we are actually trying to achieve the "isolation" approach.

Proposal

Perhaps we should have two phases for selenium tests in Jenkins. One for "isolation", and then a second phase that is WMF-specific that runs a subset of tests that we know should pass when everything is installed (e.g. skipping the default Search and CreateAccount tests). This WMF-specific config could also be re-used for a Jenkins job that targets Beta Cluster, or testwiki, for example.

For the simple case of a feature being replaced (e.g. the Search or CreateAccount page), the "Together" phase can safely disable the core test because the extension is expected to provide its own tests to cover the feature instead. And the core feature would still be covered in the "isolation" phase.

However, there is still a grey area of features that are part of core that an extension modifies in a minor way (not disabled or replaced). Such as VisualEditor's welcome dialog being added to core's plaintext page editor. It doesn't make sense for VisualEditor to have copy of the test suite for core's page editor just to disable the modal dialog in advance.

We need to decide on a general strategy here to make sure we all follow the same direction to avoid technical debt, and prevent workflow conflicts.

Thanks @Krinkle as usual you wrote a nice summary. Releng talked about it in May 2017 during our offsite, and again in May 2018 but I have lacked time to properly think about the issue.

T200976 is similar, proposing to add Scribunto as a dependency to Wikibase which would slow it down by a few minutes. Wikibase itself is added as a dependency to a few extensions which, although they have just a few tests, takes age since all Wikibase tests are run for it.

There are a few other related tasks here and there (such as extensions conflicting on parser tests). When I back from vacations I will dig into them, reuse your summary here as a basis and write a document explaining the whole problem. Then I guess put that as a RFC and we can implement the solution for each of our test suites (phpunit, parsertests, qunit, selenium).

I'm working on T218345: Investigate Selenium tests for validating event logging data, where I'd like to set a few extension specific settings needed for the tests to run:

$wgGEHelpPanelEnabled = true;
$wgGEHelpPanelLoggingEnabled = true;
$wgGEHelpPanelHelpDeskTitle = 'Main_Page';
$wgGEHelpPanelLinks = [ 'id' => 'foo', 'title' => 'Main_Page', 'text' => 'bar' ];
$wgGEHomepageEnabled = true;

Is there any way for me to do this? AFAICT the tests/browser/LocalSettings.php solution used by MobileFrontend and Minerva requires the cucumber based Selenium project.

I was surprised to see this task open. This also appears to be blocking moving our browser tests from Ruby to Node.js (T174018).
I can work around this for the time being, but certain tests are intentionally disabled on master (because they are not production ready) but need browser tests (catch 22 - the browser tests help make them production ready) ewe can't possibly migrate everything until that has happened but we have browser tests that require config changes. We need somewhere to make these config changes, even if this is in a LocalSettings.php in the CI runner itself.

Please advise how to proceed with this one.

@Jdlrobson if you're blocked on this by Release-Engineering-Team, please add it under our blocking section in https://etherpad.wikimedia.org/p/Scrum-of-Scrums

I have done exactly that. We are not blocked on this right now but will be in about 2 weeks.

Here's a concrete example, the first browser test we ported couldn't be fully ported as it depends on $wgJobRunRate being defined.. otherwise it's subject to a delay of 1 minute.

In T199939#5095838, @Jdlrobson wrote:

Here's a concrete example, the first browser test we ported couldn't be fully ported as it depends on $wgJobRunRate being defined.. otherwise it's subject to a delay of 1 minute.

Similar use cases have already come up in other projects. They have been accommodated in the form of wdio-mediawiki/RunJobs.

For an integration test that asserts behaviour across the async boundary of submitting and running of jobs, use RunJobs.run().

I found that relying on wgJobRunRate leads to tests that can be flaky and don't work consistently across different kinds of developer environments. It's also not compatible with Beta/production.

Cool! This helps this particular case, but won't help where we need to test certain features that are not yet enabled in production.

What I ended up doing here was setting the defaults to the values I need enabled for testing, which seems safe enough given that InitialiseSettings.php always sets the defaults for those particular feature flags to false. But yeah, that's not going to help if you need to change variables outside the scope of your extension, unless you have some code to load configuration variables based on the environment you're in.

Ran into this problem again. A fundamental problem with MobileFrontend is when installed it needs to be configured with a skin using wgMFDefaultSkinClass.

I can workaround it - https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/502907/ - but it's not ideal as it doesn't test what our user's really experience.

In T199939#5098977, @Jdlrobson wrote:

Cool! This helps this particular case, but won't help where we need to test certain features that are not yet enabled in production.

FWIW i couldn't get this to work in the end (https://gerrit.wikimedia.org/r/502593)

In T199939#5098977, @Jdlrobson wrote:

Cool! This helps this particular case, but won't help where we need to test certain features that are not yet enabled in production.

The pattern I recommend is for features that are merged into the master branch, to be technically available in some way by default. That seems easiest for development, testing, as well as beta testing. It could be behind a user preference, for example (which a browser test can toggle).

The server configuration can then be used to disallow the feature from being used (e.g. wgFooEnableX = false), which we'd do for production and/or beta cluster as needed until ready for wider exposure.

In https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/504543/ Vector (and I'm guessing Minerva) is not installed. How to enable other dependencies in the Jenkins CI build?

In T199939#5130664, @Jdlrobson wrote:

In https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/504543/ Vector (and I'm guessing Minerva) is not installed. How to enable other dependencies in the Jenkins CI build?

That one sounds unconnected to this task about potentially dropping tests/selenium/LocalSettings.php. wmf-quibble-vendor-mysql-hhvm-docker jobs do have Vector and its definitely loaded via a wfLoadSkin(). But indeed Minerva skin is not added, just need a task to add it to the list of gated repositories in integration/config

Adding Minerva is T202030

Will talk about this tomorrow in the meeting with @zeljkofilipin

Change 540118 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Allow uploads to support FileImporter tests

https://gerrit.wikimedia.org/r/540118

Change 540118 merged by jenkins-bot:
[integration/quibble@master] Allow uploads to support FileImporter tests

https://gerrit.wikimedia.org/r/540118

Change 541361 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] Simplify settings overrides

https://gerrit.wikimedia.org/r/541361

Change 541361 merged by jenkins-bot:
[integration/quibble@master] Simplify settings overrides

https://gerrit.wikimedia.org/r/541361

Mentioned in SAL (#wikimedia-releng) [2019-10-08T20:19:18Z] <hashar> Tag Quibble 0.0.36 @ 4b46156 # T181942 T190829 T199939 T230340 T233140 T233143

MobileFrontend and Popups have been ported to Node.js without LocalSettings.php

So until there is an easy replacement, I wanted to use this in GlobalWatchlist for T284521: Add selenium tests for GlobalWatchlist, but it appears that the tests/selenium/LocalSettings.php file is only loaded for Wikibase for the selenium tests? The only code I could find for loading such a file is

• tests/browser/LocalSettings.php https://gerrit.wikimedia.org/g/integration/config/+/9c88037a5359a4a9659f2df2e2e51a9089cc22dd/dockerfiles/quibble-stretch-bundle/mwselenium.sh#35 for cucumber tests
• tests/selenium/LocalSettings.php https://gerrit.wikimedia.org/g/integration/config/+/9c88037a5359a4a9659f2df2e2e51a9089cc22dd/jjb/mediawiki.yaml#995 for Wikibase specifically

Selenium tests were removed from Mediaviewer.