Page MenuHomePhabricator

Write unit/integration tests for Ultraviolet
Open, MediumPublic

Description

UV is currently direly in need of unit and integration tests. We expose ourselves to some very severe outages (like in Wikipedia talk:Ultraviolet § Dialog popups not opening) due to a lack of tests.

This is likely to be a long-standing effort, since there's many moving parts we have to account for.

Framework

Checkboxes will be expanded based on extra work to do along the way.

  • Migrate to Playwright
    • We currently use Puppeteer for tests. It's severely limited to Chrome and Firefox for now, which is not a good thing considering the diverse user agent base that uses UV.
  • Create isolated testing environment
    • The goal is to have reusable environment similar to the English Wikipedia that can run in an isolated environment. Few possible options for this:
      1. Current leading thought here is to use a disposable Docker image preloaded with the required pages and templates.
      2. Second thought is to use a hacked version of MediaWiki that pulls wikitext from the English Wikipedia on-demand, allowing read-only access to latest on-wiki resources. This removes having to periodically re-export required pages from the English Wikipedia, but may require extensions/some backend magic to make the API think that data actually exists (e.g. action=query may return text, but it may be missing pagelinks, etc.)
      3. Option C: Beta Cluster.
  • Make it easier to test MediaWiki features
    • A lot of the testing is probably going to be checking for pagelinks/wikitext/templates/etc. An easier way to make tests repeatable would be great.
  • Enable external use
    • Being able to use this testing suite outside of UV would benefit the userscript/tool/gadget space a lot.

Goals

  • Write unit tests for all utilities
  • Write integration tests
    • Diff icons (revert/rollback)
    • Reporting
    • Requesting page protection
    • Preferences
  • At least 60% coverage, 80% on critical systems (diff icons, reporting, protection requests)
  • No edits made on production wikis during testing
  • Test suites complete within 10 minutes (at worst)
  • (stretch) Tests work completely offline (besides possible environment setup)

Event Timeline

Chlod triaged this task as Medium priority.
Chlod added a subscriber: Remagoxer.

I can carry the Framework part (since this is quite a large endeavor), @Remagoxer has stated intention to do the latter part, but no commitments for now (so leaving them unassigned).