UV is currently direly in need of unit and integration tests. We expose ourselves to some very severe outages (like in Wikipedia talk:Ultraviolet § Dialog popups not opening) due to a lack of tests.

This is likely to be a long-standing effort, since there's many moving parts we have to account for.

Framework

Checkboxes will be expanded based on extra work to do along the way.

• Migrate to Playwright
  • We currently use Puppeteer for tests. It's severely limited to Chrome and Firefox for now, which is not a good thing considering the diverse user agent base that uses UV.
• Create isolated testing environment
  • The goal is to have reusable environment similar to the English Wikipedia that can run in an isolated environment. Few possible options for this:
    1. Current leading thought here is to use a disposable Docker image preloaded with the required pages and templates.
    2. Second thought is to use a hacked version of MediaWiki that pulls wikitext from the English Wikipedia on-demand, allowing read-only access to latest on-wiki resources. This removes having to periodically re-export required pages from the English Wikipedia, but may require extensions/some backend magic to make the API think that data actually exists (e.g. action=query may return text, but it may be missing pagelinks, etc.)
    3. Option C: Beta Cluster.
    4. Option D: Catalyst (WIP project)
• Make it easier to test MediaWiki features
  • A lot of the testing is probably going to be checking for pagelinks/wikitext/templates/etc. An easier way to make tests repeatable would be great.
• Enable external use
  • Being able to use this testing suite outside of UV would benefit the userscript/tool/gadget space a lot.

Goals

• Write unit tests for all utilities
• Write integration tests
  • Diff icons (revert/rollback)
  • Reporting
  • Requesting page protection
  • Preferences
• At least 60% coverage, 80% on critical systems (diff icons, reporting, protection requests)
• No edits made on production wikis during testing
• Test suites complete within 10 minutes (at worst)
• (stretch) Tests work completely offline (besides possible environment setup)