Page MenuHomePhabricator

Add tests for Codex components to Pixel
Closed, ResolvedPublic3 Estimated Story Points


As a Codex designer and engineer, I want to rely on automated visual regression testing to detect and prevent unwanted visual alterations resulting from applying changes to components or tokens.

Following T308447 we decided to add tests for Codex components to Pixel, the visual regression testing tool developed and currently maintained by the Web team. See T308447#8498764 for the investigation summary.

From its README:

Pixel is a visual regression tool for MediaWiki developers/QA engineers that helps you replace manual testing with automated tests that catch web ui regressions before users see them. It currently integrates BackstopJS, MediaWiki-Docker, and Docker under the hood.

Check out the hourly reports Pixel has generated at which compare the latest release branch against master, and read T302246 for the motivation behind the tool.

Timeline: Q3 2022-2023. We would like to have this in place before we release CSS-only components (T321351).

Acceptance criteria:

  • Fix the Less compiling issue described in T325237 so we can use design tokens within MediaWiki We resolved this in a temporary way via T326591, so this is no longer an issue
  • Submit a patch to Pixel to add Codex components and discuss with the Web team


  • Improve the Sandbox page so we can avoid muddying the diff when there are vertical layout or size changes
  • Improve the workflow for updating Codex code within the Sandbox
  • Add VRT to Codex itself

Event Timeline

ldelench_wmf raised the priority of this task from Medium to High.Jan 27 2023, 4:22 PM

@AnneT I added Codex to the list of reports on, where it will run the following command approximately every 30 minutes (top and bottom of each hour):

./pixel.js reference -b latest-release --group codex; ./pixel.js test --group codex

Please let me know if you run into any issues!

@nray awesome, thanks so much for all your help with this!

AnneT changed the task status from Open to In Progress.Feb 8 2023, 4:02 PM

I've added T329194 to capture further work, but this initial setup can be signed-off on