Page MenuHomePhabricator

Define JJB builder for running a subset of integration MW-Selenium tests
Closed, ResolvedPublic

Description

While we still have a lot of work to do on nodepool and isolated CI, there are a few projects/teams that have small enough (sub)sets of MW-Selenium integration tests that experimenting with a Gerrit triggered JJB builder/template seems reasonable. The recent efforts by @Jdlrobson to set up an ad hoc CI bot for per-patch tests has proven that the set of tests can be small and refined enough to be stable in this use case.

We'll have to keep the set for each project lean- and mean-ish—as much as end-to-end style tests can be—using a convention of tags and/or Cucumber profiles. A lot of the hard work of setting up MediaWiki + extension + dependencies has been done for the QUnit jobs. Let's use that as a starting point.

We'll also need to make sure that the test user account can be created at runtime using the API helper in MW-Selenium—it might worthwhile to see if we can randomize/scope the user name to allow for stable parallelization within the same test suite.

Related Objects

Event Timeline

dduvall created this task.Jun 19 2015, 12:17 AM
dduvall claimed this task.
dduvall raised the priority of this task from to Medium.
dduvall updated the task description. (Show Details)
dduvall moved this task from Inbox to In Progress on the Browser-Tests-Infrastructure board.
dduvall set Security to None.
dduvall added subscribers: Aklapper, phuedx, Jdlrobson and 9 others.

Change 219513 had a related patch set uploaded (by Dduvall):
WIP Builder for running MW-Selenium integration tests

https://gerrit.wikimedia.org/r/219513

Change 220348 had a related patch set uploaded (by Dduvall):
Integration tests to be run by CI

https://gerrit.wikimedia.org/r/220348

Change 220368 had a related patch set uploaded (by Dduvall):
Add mw-set-env-mw-selenium.sh for mw-selenium builder

https://gerrit.wikimedia.org/r/220368

Change 220368 merged by jenkins-bot:
Add mw-set-env-mw-selenium.sh for mw-selenium builder

https://gerrit.wikimedia.org/r/220368

Change 220547 had a related patch set uploaded (by Dduvall):
Set correct MW URLs for MW-Selenium

https://gerrit.wikimedia.org/r/220547

Change 220547 merged by jenkins-bot:
Set correct MW URLs for MW-Selenium

https://gerrit.wikimedia.org/r/220547

dduvall renamed this task from Experiment with JJB builder for running a subset of integration MW-Selenium tests to Define JJB builder for running a subset of integration MW-Selenium tests.Jun 26 2015, 9:52 PM

Change 219513 merged by jenkins-bot:
Builder for running MW-Selenium integration tests

https://gerrit.wikimedia.org/r/219513

Change 220348 merged by jenkins-bot:
Integration tests to be run by CI

https://gerrit.wikimedia.org/r/220348

This is up and running for MW-Selenium itself—the project is dogfooding the JJB builder with its own integration tests.[1] I've just run into a bug in the Headless gem that's preventing this from working on some slaves, but I've submitted a PR to fix it.[2][3]

[1] https://integration.wikimedia.org/ci/job/mediawiki-selenium-integration/16/
[2] https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-mw-selenium/5/console
[3] https://github.com/leonid-shevtsov/headless/pull/60

Change 222223 had a related patch set uploaded (by Dduvall):
Fix tests directory and "missing skin" errors in MW-Selenium jobs

https://gerrit.wikimedia.org/r/222223

Change 223188 had a related patch set uploaded (by Dduvall):
Support generic MW-Selenium job for MediaWiki extensions

https://gerrit.wikimedia.org/r/223188

Change 223189 had a related patch set uploaded (by Dduvall):
Generalize MW-Selenium job for MediaWiki extensions

https://gerrit.wikimedia.org/r/223189

Change 222223 abandoned by Dduvall:
Fix tests directory and "missing skin" errors in MW-Selenium jobs

Reason:
Obsoleted by Ie4636ed8fa836e7740d9d66e352472258db6935e

https://gerrit.wikimedia.org/r/222223

Change 223188 merged by jenkins-bot:
Support generic MW-Selenium job for MediaWiki extensions

https://gerrit.wikimedia.org/r/223188

Change 223189 merged by jenkins-bot:
Generalize MW-Selenium job for MediaWiki extensions

https://gerrit.wikimedia.org/r/223189

Gave it a try on the opened change for MobileFrontend https://gerrit.wikimedia.org/r/#/c/221313/ and the job happily runs something:

https://integration.wikimedia.org/ci/job/mwext-mw-selenium/6/console

Change 223691 had a related patch set uploaded (by Dduvall):
contint: Install chromedriver for running MW-Selenium tests

https://gerrit.wikimedia.org/r/223691

Change 224179 had a related patch set uploaded (by Dduvall):
Create and cleanup TMPDIR for mw-selenium builder

https://gerrit.wikimedia.org/r/224179

greg awarded a token.Jul 13 2015, 9:26 PM

Change 224179 merged by jenkins-bot:
Create and cleanup TMPDIR for mw-selenium builder

https://gerrit.wikimedia.org/r/224179

dduvall closed this task as Resolved.Jul 28 2015, 11:31 PM

There's still room for improvement in the way of scenario isolation but the MVP + video recording is working well. The capabilities were announced to qa@ and documented on mw.org.[1]

[1] https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests

Change 223691 merged by Giuseppe Lavagetto:
contint: Install chromedriver for running MW-Selenium tests

https://gerrit.wikimedia.org/r/223691