Page MenuHomePhabricator
Create Task
Maniphest T333548

Write unit/integration tests for Ultraviolet
Open, MediumPublic
Actions

Description

UV is currently direly in need of unit and integration tests. We expose ourselves to some very severe outages (like in Wikipedia talk:Ultraviolet § Dialog popups not opening) due to a lack of tests.

This is likely to be a long-standing effort, since there's many moving parts we have to account for.

Framework

Checkboxes will be expanded based on extra work to do along the way.

  • Migrate to Playwright
    • We currently use Puppeteer for tests. It's severely limited to Chrome and Firefox for now, which is not a good thing considering the diverse user agent base that uses UV.
  • Create isolated testing environment
    • The goal is to have reusable environment similar to the English Wikipedia that can run in an isolated environment. Few possible options for this:
      1. Current leading thought here is to use a disposable Docker image preloaded with the required pages and templates.
      2. Second thought is to use a hacked version of MediaWiki that pulls wikitext from the English Wikipedia on-demand, allowing read-only access to latest on-wiki resources. This removes having to periodically re-export required pages from the English Wikipedia, but may require extensions/some backend magic to make the API think that data actually exists (e.g. action=query may return text, but it may be missing pagelinks, etc.)
      3. Option C: Beta Cluster.
      4. Option D: Catalyst (WIP project)
  • Make it easier to test MediaWiki features
    • A lot of the testing is probably going to be checking for pagelinks/wikitext/templates/etc. An easier way to make tests repeatable would be great.
  • Enable external use
    • Being able to use this testing suite outside of UV would benefit the userscript/tool/gadget space a lot.

Goals

  • Write unit tests for all utilities
  • Write integration tests
    • Diff icons (revert/rollback)
    • Reporting
    • Requesting page protection
    • Preferences
  • At least 60% coverage, 80% on critical systems (diff icons, reporting, protection requests)
  • No edits made on production wikis during testing
  • Test suites complete within 10 minutes (at worst)
  • (stretch) Tests work completely offline (besides possible environment setup)

Event Timeline

Chlod created this task.Mar 30 2023, 11:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 30 2023, 11:57 AM
Chlod claimed this task.Mar 30 2023, 11:57 AM
Chlod triaged this task as Medium priority.
Chlod added a subscriber: Remagoxer.

I can carry the Framework part (since this is quite a large endeavor), @Remagoxer has stated intention to do the latter part, but no commitments for now (so leaving them unassigned).

Chlod moved this task from Needs Triaging to Backlog on the Ultraviolet board.Apr 20 2023, 1:14 AM
Chlod updated the task description. (Show Details)Apr 22 2023, 2:35 AM
Chlod updated the task description. (Show Details)Apr 17 2024, 7:59 AM
Chlod removed Chlod as the assignee of this task.Apr 17 2024, 8:03 AM

De-assigning until I can dedicate more time to this. Here's a summary of where I'm at though:

It's hard to create tests that depend on (specifically) Wikipedia. Spinning up a temporary MediaWiki instance and using that discounts a number of things: templates, extension configuration, and some particularly complicated backend configurations that make Wikipedia work. Lots of templates and modules are required on the English Wikipedia for processes and the like to just "click", and small misconfigurations can throw off UV or cause tests to fail.

In "Option A" of "Create isolated testing environment", I experimented with using scripts to build a Docker container that would include all the extensions and other services required. Unfortunately, this becomes too big to manage by a single volunteer time, so it's not worth the time. "Option B" is too technically complicated as well. "Option C" of using Beta Cluster is a good path for us to take. I was given interface administrator to explore using UV in a development environment, but was unable to continue the work since it required reworking a large section of UV to avoid cross-domain script loads (from Toolforge, specifically), which are blocked by CSP. I've just added "Option D", which is Catalyst, but there's little details about it right now — how it works, when it can be used, how to use it, and whether volunteers will be allowed access — that would make it a potential future environment.

If we happen to resume focus on this in the near future, Beta Cluster would probably be the way to go, with an eventual migration to other systems, if they happen to be better. For now, I'm inclined to keep this on hold while we work on other things that have better short-term feasibility.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL