Page MenuHomePhabricator

CirrusSearch smoke selenium tests cause failures of mediawiki-core-qunit-selenium-jessie job for extensions
Closed, ResolvedPublic

Description

Background: discovered when adding a node selenium test to WikibaseLexeme extension.

In order to run node selenium tests of the extension, we've added mediawiki-core-qunit-selenium-jessie job to extension config template (see https://gerrit.wikimedia.org/r/#/c/419479/). I've assumed that this job runs all node selenium tests of MW core, and all installed extensions. This seemed like a reasonable thing to do as a part of tests of the extension.

Our extension installs CirrusSearch as a part of the test environment, and apparently said job does not compute together with the way CirrusSearch structures its selenium tests, see the test failure: https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-selenium-jessie/16955/console.

Apparently CirrusSearch has its regular selenium tests in tests/integration, and runs them using the npm job. tests/selenium only contains smoke tests, which, according to README (https://phabricator.wikimedia.org/diffusion/ECIR/browse/master/tests/selenium/README.md), are run daily targeting beta enwiki. In other words, CirrusSearch does not even use mediawiki-core-qunit-selenium-jessie in their test pipeline. And no wonder those smoke tests fail when run targeting local wiki as part of our CI build.

So it could be that tests/selenium is not the place the extension should their "normal" selenium tests. We could have made a mistake in WikibaseLexeme, but it seemed other extensions, e.g. AdvancedSearch, or Popups have made it too.

What would be a reasonable and right thing to do here? We don't want to force CirrusSearch to change anything, clearly this structure makes sense for them. We'd prefer to use a tests/selenium as a directory to store our tests though, and also skipping CirrusSearch from the test environment is also not really an option.
Should we move our selenium tests to some other dir, not use mediawiki-core-qunit-selenium-jessie, and have them run using npm test, using something similar CirrusSearch does: https://phabricator.wikimedia.org/diffusion/ECIR/browse/master/Gruntfile.js

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptMar 19 2018, 12:21 PM

I guess @zeljkofilipin can look into it and offer some recommendation :]

WMDE-leszek updated the task description. (Show Details)Mar 19 2018, 1:05 PM

you should put tests in tests/selenium folder and use mediawiki-core-qunit-selenium-jessie job. i will disable CirrusSearch tests in mediawiki-core-qunit-selenium-jessie.

zeljkofilipin triaged this task as Normal priority.Mar 20 2018, 11:47 AM

Change 420785 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Selenium: Do not run CirrusSearch smoke tests in Jenkins

https://gerrit.wikimedia.org/r/420785

zeljkofilipin removed zeljkofilipin as the assignee of this task.Mar 21 2018, 11:20 AM

Change 420785 merged by jenkins-bot:
[mediawiki/core@master] Selenium: Do not run CirrusSearch smoke tests in Jenkins

https://gerrit.wikimedia.org/r/420785

mediawiki-core-qunit-selenium-jessie is no longer failing because of CirrusSearch tests: mediawiki-core-qunit-selenium-jessie/17626.

For ERROR: Cannot find module 'nodemw' failures see: T181284, T190307 and T190426.

This is resolved, right? Is there anything left to do?

WMDE-leszek closed this task as Resolved.Apr 5 2018, 3:28 PM
WMDE-leszek claimed this task.

This seems fixed indeed, thanks @zeljkofilipin! We're encountering another issues with node selenium tests, but I'll file a separate ticket about this in a bit.

Change 434011 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[integration/config@master] Do not run composer in WikibaseLexeme's node selenium job

https://gerrit.wikimedia.org/r/434011