Page MenuHomePhabricator
Create Task
Maniphest T222200

Run browser tests as part of "npm test" for wikidata/query/gui in CI
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

We want to have the browser tests running in CI so that we don't miss tests breaking.
T267393 (also being tackled right now) makes sure that these tests pass locally.

Note: Lots of the discussion in the comments are discussing issues that are long gone, only comments in 2021+ are relevant now
Also: The current attached Gerrit changes may be addressing issues that are no longer relevant

Link to the repository: https://gerrit.wikimedia.org/r/admin/repos/wikidata/query/gui
Link to the repository of the jenkins config: https://gerrit.wikimedia.org/g/integration/config/

Acceptance Criteria

  • Browser tests run in CI for the wikidata/query/gui repository

Details

ProjectBranchLines +/-Subject
wikidata/query/guimaster+3 -3Rename separate grunt task from 'browser_test' to 'test_all'
wikidata/query/guimaster+305 -197WIP: Run wdio without selenium-standalone
wikidata/query/guimaster+19 -127Run browser tests in npm test, but don't run webdriver
integration/configmaster+7 -1Use {name}-node10-browser-webdriver-docker for wikidata/query/gui
integration/configmaster+20 -0Add {name}-node10-browser-webdriver-docker
integration/configmaster+9 -1Change Dockerfile and wikidata.yml to support running browser tests for the query service gui
wikidata/query/guimaster+28 -14Make browser tests run in CI
Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
WMDE-leszek added subscribers: zeljkofilipin, hashar.Apr 30 2019, 4:22 PM

@hashar @zeljkofilipin this seems to beyond our (WMDE) realm. We're happy to provide more context/information, but we'd appreciate advise/help with getting this actually resolved.

zeljkofilipin triaged this task as Medium priority.May 2 2019, 2:03 PM
zeljkofilipin added a project: User-zeljkofilipin.
zeljkofilipin moved this task from Backlog 🪒 to Next 💉 on the User-zeljkofilipin board.
zeljkofilipin added a comment.May 10 2019, 10:22 AM
  • I will be at Wikimedia-Hackathon-2019 next week, if anybody wants to pair on this.
  • The convention is to have Selenium tests (including wdio.conf.js) in tests/selenium, if possible. See Selenium/Node.js#write-tests for examples.
  • I don't think CI supports running Selenium tests in Firefox (wdio.conf.js#33). All other repositories use Chrome.
  • There is no need for selenium-standalone (Gruntfile.js#273). Use Chromedriver instead. It should be available in CI.
zeljkofilipin moved this task from Next 💉 to Waiting 🏹 on the User-zeljkofilipin board.May 10 2019, 10:22 AM
noarave added a comment.May 13 2019, 7:52 AM

@zeljkofilipin I will be at the Hackathon and would gladly pair on this. I arrive on Fri 17.05.

zeljkofilipin added a project: Wikimedia-Hackathon-2019.May 13 2019, 10:26 AM
zeljkofilipin moved this task from Backlog to Projects on the Wikimedia-Hackathon-2019 board.
zeljkofilipin moved this task from Waiting 🏹 to Deep work 🌊 on the User-zeljkofilipin board.May 18 2019, 11:47 AM
zeljkofilipin moved this task from Deep work 🌊 to Documentation 📚 on the User-zeljkofilipin board.May 18 2019, 12:50 PM
gerritbot added a comment.May 19 2019, 12:14 PM

Change 511308 had a related patch set uploaded (by Noa wmde; owner: Noa wmde):
[integration/config@master] Change Dockerfile and wikidata.yml to support running browser tests for the query service gui

https://gerrit.wikimedia.org/r/511308

gerritbot added a project: Patch-For-Review.May 19 2019, 12:15 PM
gerritbot added a comment.May 19 2019, 12:22 PM

Change 511309 had a related patch set uploaded (by Noa wmde; owner: Noa wmde):
[wikidata/query/gui@master] Rename separate grunt task from 'browser_test' to 'test_all'

https://gerrit.wikimedia.org/r/511309

noarave mentioned this in rWDQG27652e6453cc: Rename separate grunt task from 'browser_test' to 'test_all'.May 19 2019, 12:30 PM
hashar added a subscriber: Smalyshev.May 23 2019, 1:02 PM

Here is an explanation of the issue @Smalyshev encountered on T219749 as I posted it on https://gerrit.wikimedia.org/r/#/c/integration/config/+/511308/

The failure comes from the CI tests for wikidata/query/rdf which are run by the Maven test suite. The job doing that is wikidata-query-rdf-maven-java8-docker which really just ship java8 and then runs maven. There is no chromedriver being spawned.

The wikidata/query/rdf test suite then run the GUI test suite using npm, but since the gui code uses wdio/chromedriver and the maven image does not have it ... The suite fails to connect to chromedriver ( ECONNREFUSED 127.0.0.1:4444 ).

I am not sure whether the GUI test should be run as part of testing wikidata/query/rdf . Probably only a subset of them should?

For wikidata/query/gui, the CI job runs npm test which in turns seems to invoke grunt test. So I guess it is all about adding wdio task to the list of tasks executed when running grunt test. Specially, revert https://gerrit.wikimedia.org/r/#/c/wikidata/query/gui/+/506666/

BUT, in wikidata/query/rdf, as soon as the GUI submodule is updated, the test will fail since they are run with maven and the container does not have chromedriver nor is it intended to run wdio test. My big question is: do you actually need to run the GUI npm test when testing wikidata/query/rdf ? That seems to just rerun tests that already ran perfectly fine in the upstream repo wikidata/query/gui.

awight added a subscriber: awight.May 23 2019, 1:06 PM

This might also be useful:
T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests
I have a patch for review which will run browser tests for each extension separately, which makes it possible to run custom setup scripts and install local npm dependencies. This seems like a simpler solution than spinning up the headless browser in another phase of testing. Or maybe I'm misunderstanding why you want the workaround here--would you mind explaining why it's desirable to run browser tests along with npm test?

noarave mentioned this in rWDQGb5558fd4cd2a: Rename separate grunt task from 'browser_test' to 'test_all'.May 27 2019, 8:28 AM
zeljkofilipin moved this task from Documentation 📚 to Backlog 🪒 on the User-zeljkofilipin board.May 31 2019, 12:40 PM
gerritbot added a comment.Nov 28 2019, 1:35 PM

Change 511309 abandoned by Hashar:
Rename separate grunt task from 'browser_test' to 'test_all'

https://gerrit.wikimedia.org/r/511309

gerritbot added a comment.Nov 28 2019, 1:36 PM

Change 511309 restored by Hashar:
Rename separate grunt task from 'browser_test' to 'test_all'

Reason:
Oops, I wanted to abandon another change sorry.

https://gerrit.wikimedia.org/r/511309

Addshore moved this task from Incoming to Blocked / Waiting / External on the Wikidata-Campsite board.Jan 16 2020, 11:33 PM
Addshore added a subscriber: Addshore.

The failure comes from the CI tests for wikidata/query/rdf which are run by the Maven test suite. The job doing that is wikidata-query-rdf-maven-java8-docker which really just ship java8 and then runs maven. There is no chromedriver being spawned

Soon the UI will not be bundled as part of the wdqs backend java stuff.
does that make all of this easier / eliminate some problems?
See T241291

If not it'd be good to get a summary / overview of what the state of everything is here!

Addshore added a comment.Jan 27 2020, 3:32 PM
In T222200#5811481, @Addshore wrote:

Soon the UI will not be bundled as part of the wdqs backend java stuff.

This is now true.
The gui is no longer bundled with the java stuff.

Lucas_Werkmeister_WMDE mentioned this in T267393: Cannot run wikidata-query-gui browser tests.Nov 6 2020, 11:36 AM
WMDE-leszek moved this task from Blocked / Waiting / External to Prioritized Wikidata Tech Backlog (prioritised from top to bottom) on the Wikidata-Campsite board.Nov 17 2020, 1:13 PM
Addshore added a project: wdwb-tech (legacy-backlog).Jan 22 2021, 11:06 PM
Addshore moved this task from Incoming to 3. Fill ins (-I -E) on the wdwb-tech (legacy-backlog) board.Jan 22 2021, 11:10 PM
Addshore moved this task from 3. Fill ins (-I -E) to 1. Quick Wins (+I -E) on the wdwb-tech (legacy-backlog) board.
Addshore renamed this task from Run browser tests as part of "npm test" of wikidata/query/gui to Run browser tests as part of "npm test" of wikidata/query/gui in CI.Jan 26 2021, 1:18 PM
Addshore renamed this task from Run browser tests as part of "npm test" of wikidata/query/gui in CI to Run browser tests as part of "npm test" for wikidata/query/gui in CI.Jan 26 2021, 1:44 PM
Addshore updated the task description. (Show Details)
Addshore updated the task description. (Show Details)
Addshore removed a subscriber: • alaa_wmde.
Addshore updated the task description. (Show Details)Feb 9 2021, 1:08 PM
Addshore updated the task description. (Show Details)
darthmon_wmde updated the task description. (Show Details)Feb 10 2021, 1:13 PM
darthmon_wmde set the point value for this task to 5.
darthmon_wmde moved this task from Prioritized Wikidata Tech Backlog (prioritised from top to bottom) to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.
darthmon_wmde edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Feb 10 2021, 1:15 PM
noarave added a comment.Feb 10 2021, 2:15 PM

Task inspection note: we probably want to start with adding the browser_test to the test script in package.jsonand see what happens

Lucas_Werkmeister_WMDE claimed this task.Feb 18 2021, 4:57 PM
Lucas_Werkmeister_WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
gerritbot added a comment.Feb 18 2021, 5:01 PM

Change 665124 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[wikidata/query/gui@master] Add wdio to grunt test task

https://gerrit.wikimedia.org/r/665124

Lucas_Werkmeister_WMDE added a comment.Feb 18 2021, 5:11 PM
In T222200#6818652, @noarave wrote:

Task inspection note: we probably want to start with adding the browser_test to the test script in package.jsonand see what happens

Tried it, doesn’t work (no Java installed in the container). We’ll probably want to switch the CI job from node10-test to node10-test-browser (or add a separate CI job using that container which only runs the browser tests?), and then either remove selenium-standalone completely or disable it in CI, in order to use the chromedriver provided by the container, as recommended in T222200#5172117.

Lucas_Werkmeister_WMDE added a comment.Feb 19 2021, 11:51 AM

Wait, nevermind, the job already runs on node10-test-browser. Not sure why wdio fails with an ECONNREFUSED error against a random port, then…

Lucas_Werkmeister_WMDE removed Lucas_Werkmeister_WMDE as the assignee of this task.Feb 19 2021, 5:31 PM
Lucas_Werkmeister_WMDE moved this task from Doing to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Lucas_Werkmeister_WMDE added a subscriber: Lucas_Werkmeister_WMDE.

I’m at a loss. I tried removing selenium-standalone completely, but wdio still fails with the same error, “connection refused” on a seemingly random port number. Even with --logLevel=trace, wdio prints just a handful of messages, not enough for me to figure out why it’s trying to connect to that port. (The port doesn’t seem to match any environment variable.) I tried bringing our wdio.conf.js closer to other repositories, but to no avail. I don’t see any other significant differences between how we launch wdio and how other repositories do it.

Unassigning myself since I’ll be working on Wikibase Service Migration starting next week. I hope someone else can figure out what’s going on here… you can see all the things I tried on the Gerrit change linked above.

• Tonina_Zhelyazkova_WMDE claimed this task.Feb 22 2021, 10:54 AM
• Tonina_Zhelyazkova_WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
gerritbot added a comment.Feb 22 2021, 11:43 AM

Change 666094 had a related patch set uploaded (by Tonina Zhelyazkova; owner: Tonina Zhelyazkova):
[wikidata/query/gui@master] Make browser tests run in CI

https://gerrit.wikimedia.org/r/666094

awight added a comment.Feb 22 2021, 1:46 PM

Quibble runs some services in the background, which you would also need in order to run browser tests. Specifically I think you're missing:

  • xvfb in order to run headless, like xvfb :94 -screen 0 1280x1024x24 -nolisten tcp -nolisten unix
  • chromedriver, either the npm driver or standalone: chromedriver --port=4444 --url-base=/wd/hub

It might also be possible to use Quibble as the test runner, like quibble --run selenium, for example in this template, but I'm not sure whether anyone has tried this outside of mediawiki-extensions-* yet?

Task inspection note: we probably want to start with adding the browser_test to the test script in package.json and see what happens

This sounds slightly nasty to me, don't you sometimes want to run just the simple tests locally? It's fine to have a separate script target and your CI configuration can call both npm run test and npm run selenium-test. But if you decide you really do want to merge the test suites, that should work too.

zeljkofilipin moved this task from Backlog 🪒 to Deep work 🌊 on the User-zeljkofilipin board.Feb 22 2021, 2:29 PM
gerritbot added a comment.Feb 22 2021, 2:56 PM

Change 666138 had a related patch set uploaded (by Tonina Zhelyazkova; owner: Tonina Zhelyazkova):
[wikidata/query/gui@master] Make browser tests run in CI

https://gerrit.wikimedia.org/r/666138

Addshore added a comment.Feb 23 2021, 2:47 PM

We could also look at just running chromedriver in a docker container as part of the CI job that is defined for query-gui.
So the job would be 1) run chromedrive 2) run tests 3) stop chromedriver?
Would need to see how possible @hashar thinks this would be but I don't see reasons that it shouldnt work?

gerritbot added a comment.Feb 23 2021, 3:03 PM

Change 666368 had a related patch set uploaded (by Awight; owner: Awight):
[wikidata/query/gui@master] [WIP] experiment with browser testing

https://gerrit.wikimedia.org/r/666368

awight added a comment.Feb 23 2021, 3:04 PM

I mashed together some bits laying around and got the above patch to pass. The only big issue is the grunt-merge-i18n package coming from github, which breaks due to missing ssh. Otherwise, I think this is usable.

• Tonina_Zhelyazkova_WMDE removed • Tonina_Zhelyazkova_WMDE as the assignee of this task.Feb 23 2021, 3:09 PM
• Tonina_Zhelyazkova_WMDE added a subscriber: • Tonina_Zhelyazkova_WMDE.

Moving to Review, so people can take a look at Adam W's patch. Thanks @awight

• Tonina_Zhelyazkova_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 23 2021, 3:10 PM
awight added a comment.Feb 23 2021, 3:14 PM

Notes:

  • The wdio-chromedriver-service seems better from a distance, but the antagonist seems to be in the details. See this commit for some notes about why the approach was abandoned in mw-core.
  • The container already includes a /run-with-xvfb.sh, ideally we can drop ./selenium.sh and set the container entrypoint to start the services, then run npm run browser_test directly.
  • I guess the simplest fix for the grunt-merge-i18n dependency is to upload it to npm?
zeljkofilipin added a comment.Feb 23 2021, 4:45 PM

Looks like @awight's latest patch makes the wdio tests run in CI.

I find some of my comments from 2019 still stand (T222200#5172117). I'll quote relevant parts:

In T222200#5172117, @zeljkofilipin wrote:
  • The convention is to have Selenium tests (including wdio.conf.js) in tests/selenium, if possible. See Selenium/Node.js#write-tests for examples.
  • There is no need for selenium-standalone (Gruntfile.js#273). Use Chromedriver instead. It should be available in CI.
  • Is there a reason this repository doesn't have Selenium tests in tests/selenium?
  • Is there a reason this repository uses selenium-standalone instead of chromedriver?
awight added a comment.Feb 23 2021, 4:52 PM
In T222200#6853286, @zeljkofilipin wrote:
  • Is there a reason this repository uses selenium-standalone instead of chromedriver?

I think we can remove that dependency if we merge my patch using standalone chromedriver from the container.

zeljkofilipin moved this task from Deep work 🌊 to Watching 📺 on the User-zeljkofilipin board.Feb 23 2021, 5:35 PM
noarave moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 24 2021, 10:18 AM
awight added a comment.Feb 24 2021, 12:23 PM

The latest patchset drops selenium-standalone.

gerritbot added a comment.Feb 24 2021, 4:07 PM

Change 666138 abandoned by Tonina Zhelyazkova:
[wikidata/query/gui@master] Make browser tests run in CI

Reason:

https://gerrit.wikimedia.org/r/666138

noarave moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 26 2021, 11:10 AM
gerritbot added a comment.Mar 1 2021, 3:25 PM

Change 667623 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] Add {name}-node10-browser-webdriver-docker

https://gerrit.wikimedia.org/r/667623

gerritbot added a comment.Mar 1 2021, 3:26 PM

Change 511308 abandoned by Addshore:
[integration/config@master] Change Dockerfile and wikidata.yml to support running browser tests for the query service gui

Reason:

https://gerrit.wikimedia.org/r/511308

noarave moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 2 2021, 8:52 AM
Addshore moved this task from 1. Quick Wins (+I -E) to Active on the wdwb-tech (legacy-backlog) board.Mar 2 2021, 8:59 AM
gerritbot added a comment.Mar 3 2021, 11:25 AM

Change 668040 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] Add {name}-node10-browser-webdriver-docker

https://gerrit.wikimedia.org/r/668040

gerritbot added a comment.Mar 3 2021, 11:33 AM

Change 668040 merged by jenkins-bot:
[integration/config@master] Add {name}-node10-browser-webdriver-docker

https://gerrit.wikimedia.org/r/668040

gerritbot added a comment.Mar 3 2021, 11:39 AM

Change 667623 merged by jenkins-bot:
[integration/config@master] Use {name}-node10-browser-webdriver-docker for wikidata/query/gui

https://gerrit.wikimedia.org/r/667623

gerritbot added a comment.Mar 3 2021, 11:43 AM

Change 666368 had a related patch set uploaded (by Addshore; owner: Awight):
[wikidata/query/gui@master] [WIP] experiment with browser testing

https://gerrit.wikimedia.org/r/666368

Addshore claimed this task.Mar 3 2021, 12:15 PM
Addshore moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

https://gerrit.wikimedia.org/r/c/wikidata/query/gui/+/666368 is the thing needing review

Restricted Application added a project: User-Addshore. · View Herald TranscriptMar 3 2021, 12:15 PM
gerritbot added a comment.Mar 5 2021, 8:27 AM

Change 666368 merged by jenkins-bot:
[wikidata/query/gui@master] Run browser tests in npm test, but don't run webdriver

https://gerrit.wikimedia.org/r/666368

Jakob_WMDE mentioned this in rWDQG4fa380df41a2: Run browser tests in npm test, but don't run webdriver.Mar 5 2021, 8:35 AM
Jakob_WMDE moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 5 2021, 10:17 AM
Addshore closed this task as Resolved.Mar 5 2021, 2:41 PM
gerritbot added a comment.Mar 5 2021, 2:42 PM

Change 665124 abandoned by Addshore:
[wikidata/query/gui@master] WIP: Run wdio without selenium-standalone

Reason:
Task is now resolved

https://gerrit.wikimedia.org/r/665124

gerritbot added a comment.Mar 5 2021, 2:42 PM

Change 511309 abandoned by Addshore:
[wikidata/query/gui@master] Rename separate grunt task from 'browser_test' to 'test_all'

Reason:
Task is now resolved

https://gerrit.wikimedia.org/r/511309

Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL