Page MenuHomePhabricator

Create and run a suite of end-to-end tests for the Wikimedia environment
Open, Needs TriagePublic

Description

Problem:
We need tests that ensure we don't break the Wikimedia setup, including the various extensions, backend services, and cross-wiki interactions we have.

Context:
The suite of api integration tests included with MediaWiki core is designed to run locally in a development environment (docker-compose), and against proposed changes during code review (CI/Jenkins). They are written to work against a vanilla configuration (with DevelopmentSessings enabled).

Things that cannot be tested against a vanilla setup include:

  • language links (because no language interwiki prefixes are defined)
  • remote image repo
  • cirrus search
  • change propagation
  • wikidata includes
  • authorization via oauth
  • etc

Solution:
Create a suite of tests designed to run against the beta cluster on labs. Run them periodically (perhaps every time we pull the latest version of the code, which is every ten minutes or so).

This test suite should live in it's own repo. It's Wikimedia-Specific.

Complication:

  • In order to run these test, we have to share at least one secret ($wgSecret) between the test runner's environment and the wiki.
  • While developing tests, it would be useful to be able to run them from the local machine, against the remote testing environment. How would sharing the secret work in this scenario? Something like a temporary bot password might work, but how would the developer acquire one?
  • Tests must be written defensively in a way that allows them to reliably function while other users (humans, selenium tests, etc) interact with the same wiki instances. For the most part, this can be achieved by randomizing page titles and user names.
  • It's unclear who would receive notification of failures. Ideally, the authors (teams?) of patches merged after the tests were last green would receive them. This may not be trivial to do, though.

Event Timeline

daniel created this task.Mar 27 2020, 1:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 27 2020, 1:33 PM
Anomie removed a subscriber: Anomie.Mar 27 2020, 1:53 PM
daniel updated the task description. (Show Details)Apr 1 2020, 4:14 PM
daniel updated the task description. (Show Details)
Jdforrester-WMF added a subscriber: Jdforrester-WMF.

Happy to help QTE lead on this work, if that's what you want to do.

daniel added a comment.EditedApr 3 2020, 8:40 PM

Happy to help QTE lead on this work, if that's what you want to do.

I think the main blocker is the shared secret. Do you have thoughts on that?

daniel updated the task description. (Show Details)Apr 3 2020, 8:42 PM
zeljkofilipin added a comment.EditedApr 7 2020, 3:07 PM
  • While developing tests, it would be useful to be able to run them from the local machine, against the remote testing environment. How would sharing the secret work in this scenario? Something like a temporary bot password might work, but how would the developer acquire one?

In Selenium context, every developer has a test account at beta cluster. Jenkins has it's own account. The framework uses the value of MW_SERVER environment variable for target site.

Documentation on how to run Selenium tests from developer machine targeting beta cluster:

  • Tests must be written defensively in a way that allows them to reliably function while other users (humans, selenium tests, etc) interact with the same wiki instances. For the most part, this can be achieved by randomizing page titles and user names.

That is what Selenium tests do. We didn't have any problems in years.

  • It's unclear who would receive notification of failures. Ideally, the authors (teams?) of patches merged after the tests were last green would receive them. This may not be trivial to do, though.

We didn't solve that problem for Selenium. Since the tests run daily, there would be many patches and authors to notify. For each repository we have a contact person: