Page MenuHomePhabricator

Tests should not rely on the existence of specific pages (be wiki agnostic)
Closed, ResolvedPublic

Description

On MobileFrontend there is a newly added test that expects the Barack Obama article to be setup in a certain way.

It fails on mine as my Barack Obama doesn't have an external links directory.
Ideally I would like to see QA tests setup pages e.g. QA_TestPage1, QA_TestPage2 (or maybe a QA namespace might make sense here) for the purpose of the test or at least be customisable to point to different pages..

Feature: Validate External Links

Scenario: External Links resolve                       # features/external_links.feature:3
  Given I am on the Barack_Obama article               # features/step_definitions/external_links_steps.rb:1
  When I expand External Links Section                 # features/step_definitions/external_links_steps.rb:5
    timed out after 5 seconds, waiting for {:id=>"External_links", :tag_name=>"span"} to become present (Watir::Wait::TimeoutError)
    ./features/step_definitions/external_links_steps.rb:6:in `/^I expand External Links Section$/'
    features/external_links.feature:5:in `When I expand External Links Section'
  And I click on the White House official website link # features/step_definitions/external_links_steps.rb:9
  Then I receive White House official website page     # features/step_definitions/external_links_steps.rb:13

Version: wmf-deployment
Severity: normal

Details

Reference
bz56057

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:27 AM
bzimport added a project: Quality-Assurance.
bzimport set Reference to bz56057.
bzimport added a subscriber: Unknown Object (MLST).

This is by design.

Browser tests in the public repo are intended to use the beta cluster, test2wiki, and/or production as targets, and we are in a position to manage the contents of those test environments. We put this in place from the very beginning to a) avoid the "works on my machine" problems b) keep the setup/teardown overhead to a minimum and c) run the tests in a complex environment that approximates production as closely as possible.

We have a few tests that create new pages where it makes sense to test creating new pages. There is no reasonable mechanism by which to delete pages in Mediawiki.

So, if you want a suite of tests to run against a local Mediawiki environment, you could do a number of things:

  • maintain a branch of tests that do not depend on particular test data being in place
  • at runtime, swap out the existing foo_page.rb files for similar files that point to the URL of your choice, override the browser.url variable, or otherwise alter the existing value for e.g. Barack_Obama
  • maintain a minimal set of test data in your local env for the tests to pass without alteration

I agree that each test should make sure everything it needs exists and create users, pages, files (...) if they already do not exist.

If deleting pages is hard, that is a separate problem that should be fixed.

Duplicate of bug #60347?

Change 125008 had a related patch set uploaded by Jdlrobson:
QA: WIP Make the test environment agnostic of existing pages

https://gerrit.wikimedia.org/r/125008

Change 125857 had a related patch set uploaded by Jdlrobson:
QA: Make the test environment agnostic of existing pages #2

https://gerrit.wikimedia.org/r/125857

Change 125859 had a related patch set uploaded by Jdlrobson:
QA: Remove dependency on Duel Masters article

https://gerrit.wikimedia.org/r/125859

Change 125861 had a related patch set uploaded by Jdlrobson:
QA: Remove dependency on "Foo bar" article

https://gerrit.wikimedia.org/r/125861

Change 125864 had a related patch set uploaded by Jdlrobson:
QA: Remove dependency on Barack Obama article

https://gerrit.wikimedia.org/r/125864

Change 125008 merged by jenkins-bot:
QA: Remove dependency on San Francisco article

https://gerrit.wikimedia.org/r/125008

Change 125857 merged by jenkins-bot:
QA: Make the test environment agnostic of existing pages #2

https://gerrit.wikimedia.org/r/125857

Change 125859 merged by jenkins-bot:
QA: Remove dependency on Duel Masters article

https://gerrit.wikimedia.org/r/125859

Change 125861 merged by jenkins-bot:
QA: Remove dependency on "Foo bar" article

https://gerrit.wikimedia.org/r/125861

Can someone review https://gerrit.wikimedia.org/r/125864 - getting it merged will resolve this bug as it will mean MobileFrontend has no page specific code.

Change 125864 merged by jenkins-bot:
QA: Remove dependency on Barack Obama article

https://gerrit.wikimedia.org/r/125864