Page MenuHomePhabricator

Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature
Open, NormalPublic

Description

When running Selenium tests locally or on CI on a per patchset basis, we went with introducing a way for extension to use different settings. Many projects do not need extra settings at all, we should look at all tests/selenium/LocalSettings.php and possibly have sane default settings that would render it useless. That might not be achievable though.

Examples:

StatusProjectSettingDescription
MultimediaViewer$wgUseInstantCommons = true;Probably has a test for instant commons but we do not want it enabled by default in MediaWiki
DonePopups$wgUsejQueryThree = false;That setting has been removed from MediaWiki
ORESvariousPoints to testwiki which might not exist, define models
VisualEditor$wgVisualEditorShowBetaWelcome = false;Disable the welcome menu which breaks MediaWiki core Selenium tests
MobileFrontendvariousSet a beta mode, wikibase settings, hook for an es interwiki
FileImporter$wgEnableUploads = true;The user must be allowed to upload for this extension to work.

Related, a discussion @zeljkofilipin and @hashar had a few months ago about PHPUnit tests: https://docs.google.com/drawings/d/1ebVAFqM80Me7HCiLV-JdAkNCh3unLCGsuNqRcgiGtEI/edit?usp=sharing

Related Objects

Event Timeline

hashar created this task.Jul 18 2018, 7:41 PM
Restricted Application added subscribers: APerson, Aklapper. · View Herald Transcript

Change 446823 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/extensions/Popups@master] QA: Selenium no more needs wgUsejQueryThree

https://gerrit.wikimedia.org/r/446823

Change 446823 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] QA: Selenium no more needs wgUsejQueryThree

https://gerrit.wikimedia.org/r/446823

Krinkle updated the task description. (Show Details)Jul 19 2018, 8:50 PM
VisualEditor$wgVisualEditorShowBetaWelcome = false;Disable the welcome menu which breaks MediaWiki core Selenium tests

If I recall correctly, the VisualEditor welcome dialog can also be hidden by others means (cookie, user preference, and url parameter). I'm guessing the problem is not with VisualEditor's own tests (which indeed, could use cookies or url parameters), but the problem is with MediaWiki core's selenium tests not passing when VE is installed?

That raises an important question: Do we want to test features in isolation, or test (shared gate extension) features together?

Isolation

If we test features in isolation, this means core tests run without extensions installed, and extension tests run with only that extension (+dependencies) installed.

Testing in isolation, means we can test any core feature, even those replaced or modified by an extension. Extensions that replace the "Search" or "Create account" page can be tested, and also the core default version.

Together

If we test the result of installing everything together, then some core features cannot be tested, and also, extensions cannot test all certain features.

  • An extension that modifies or adds to a core feature, such as the welcome dialog from VisualEditor, do we disable that so that the core test passes? What if VisualEditor wants to test that the welcome dialog works? If we disable it, that also means the test no longer represents the normal configuration of the extension.
  • A change that breaks a core feature, but its breakage is repaired/masked by an extension – how do we catch this regression from MediaWiki core's tests?
Currently

We currently test everything together. But, at the same time, we use selenium/LocalSettings.php to "disable" interference from extensions with core features. This sounds like we are actually trying to achieve the "isolation" approach.

Proposal

Perhaps we should have two phases for selenium tests in Jenkins. One for "isolation", and then a second phase that is WMF-specific that runs a subset of tests that we know should pass when everything is installed (e.g. skipping the default Search and CreateAccount tests). This WMF-specific config could also be re-used for a Jenkins job that targets Beta Cluster, or testwiki, for example.

For the simple case of a feature being replaced (e.g. the Search or CreateAccount page), the "Together" phase can safely disable the core test because the extension is expected to provide its own tests to cover the feature instead. And the core feature would still be covered in the "isolation" phase.

However, there is still a grey area of features that are part of core that an extension modifies in a minor way (not disabled or replaced). Such as VisualEditor's welcome dialog being added to core's plaintext page editor. It doesn't make sense for VisualEditor to have copy of the test suite for core's page editor just to disable the modal dialog in advance.

We need to decide on a general strategy here to make sure we all follow the same direction to avoid technical debt, and prevent workflow conflicts.

hashar updated the task description. (Show Details)Jul 23 2018, 9:23 AM
hashar added a comment.Aug 3 2018, 9:12 AM

Thanks @Krinkle as usual you wrote a nice summary. Releng talked about it in May 2017 during our offsite, and again in May 2018 but I have lacked time to properly think about the issue.

T200976 is similar, proposing to add Scribunto as a dependency to Wikibase which would slow it down by a few minutes. Wikibase itself is added as a dependency to a few extensions which, although they have just a few tests, takes age since all Wikibase tests are run for it.

There are a few other related tasks here and there (such as extensions conflicting on parser tests). When I back from vacations I will dig into them, reuse your summary here as a basis and write a document explaining the whole problem. Then I guess put that as a RFC and we can implement the solution for each of our test suites (phpunit, parsertests, qunit, selenium).

This comment was removed by Krinkle.
JTannerWMF moved this task from To Triage to Freezer on the VisualEditor board.Feb 12 2019, 5:14 PM

I'm working on T218345: Investigate Selenium tests for validating event logging data, where I'd like to set a few extension specific settings needed for the tests to run:

$wgGEHelpPanelEnabled = true;
$wgGEHelpPanelLoggingEnabled = true;
$wgGEHelpPanelHelpDeskTitle = 'Main_Page';
$wgGEHelpPanelLinks = [ 'id' => 'foo', 'title' => 'Main_Page', 'text' => 'bar' ];
$wgGEHomepageEnabled = true;

Is there any way for me to do this? AFAICT the tests/browser/LocalSettings.php solution used by MobileFrontend and Minerva requires the cucumber based Selenium project.

I was surprised to see this task open. This also appears to be blocking moving our browser tests from Ruby to Node.js (T174018).
I can work around this for the time being, but certain tests are intentionally disabled on master (because they are not production ready) but need browser tests (catch 22 - the browser tests help make them production ready) ewe can't possibly migrate everything until that has happened but we have browser tests that require config changes. We need somewhere to make these config changes, even if this is in a LocalSettings.php in the CI runner itself.

Please advise how to proceed with this one.

@Jdlrobson if you're blocked on this by Release-Engineering-Team, please add it under our blocking section in https://etherpad.wikimedia.org/p/Scrum-of-Scrums

I have done exactly that. We are not blocked on this right now but will be in about 2 weeks.

Here's a concrete example, the first browser test we ported couldn't be fully ported as it depends on $wgJobRunRate being defined.. otherwise it's subject to a delay of 1 minute.

Here's a concrete example, the first browser test we ported couldn't be fully ported as it depends on $wgJobRunRate being defined.. otherwise it's subject to a delay of 1 minute.

Similar use cases have already come up in other projects. They have been accommodated in the form of wdio-mediawiki/RunJobs.

For an integration test that asserts behaviour across the async boundary of submitting and running of jobs, use RunJobs.run().

I found that relying on wgJobRunRate leads to tests that can be flaky and don't work consistently across different kinds of developer environments. It's also not compatible with Beta/production.

Cool! This helps this particular case, but won't help where we need to test certain features that are not yet enabled in production.

kostajh added a comment.EditedApr 9 2019, 7:56 PM

What I ended up doing here was setting the defaults to the values I need enabled for testing, which seems safe enough given that InitialiseSettings.php always sets the defaults for those particular feature flags to false. But yeah, that's not going to help if you need to change variables outside the scope of your extension, unless you have some code to load configuration variables based on the environment you're in.

Ran into this problem again. A fundamental problem with MobileFrontend is when installed it needs to be configured with a skin using wgMFDefaultSkinClass.

I can workaround it - https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/502907/ - but it's not ideal as it doesn't test what our user's really experience.

Cool! This helps this particular case, but won't help where we need to test certain features that are not yet enabled in production.

FWIW i couldn't get this to work in the end (https://gerrit.wikimedia.org/r/502593)

zeljkofilipin triaged this task as Normal priority.Apr 11 2019, 11:57 AM

Cool! This helps this particular case, but won't help where we need to test certain features that are not yet enabled in production.

The pattern I recommend is for features that are merged into the master branch, to be technically available in some way by default. That seems easiest for development, testing, as well as beta testing. It could be behind a user preference, for example (which a browser test can toggle).

The server configuration can then be used to disallow the feature from being used (e.g. wgFooEnableX = false), which we'd do for production and/or beta cluster as needed until ready for wider exposure.

In https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/504543/ Vector (and I'm guessing Minerva) is not installed. How to enable other dependencies in the Jenkins CI build?

In https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/504543/ Vector (and I'm guessing Minerva) is not installed. How to enable other dependencies in the Jenkins CI build?

That one sounds unconnected to this task about potentially dropping tests/selenium/LocalSettings.php. wmf-quibble-vendor-mysql-hhvm-docker jobs do have Vector and its definitely loaded via a wfLoadSkin(). But indeed Minerva skin is not added, just need a task to add it to the list of gated repositories in integration/config

Adding Minerva is T202030

Will talk about this tomorrow in the meeting with @zeljkofilipin

awight updated the task description. (Show Details)Wed, Sep 11, 10:01 AM