## Summary
As a Codex designer and engineer, I want to be able to rely on automated visual regression testing to detect and prevent unwanted visual alterations resulting from applying changes to components or tokens.
### Acceptance Criteria
[x] Connect with Readers Web to review existing visual regression tools and capabilities (+ @nray) to see if this would be a viable path forward for Codex components
---
## Findings
We have two options, and may want to implement them both at some point:
1. Add tests for Codex components to [[ https://github.com/wikimedia/pixel | Pixel ]], the visual regression testing tool developed and currently maintained by the Web team (although ownership may transfer to QTE)
2. Add visual regression testing to Codex itself
---
### Add testing of Codex components to Pixel
Summary: Regular visual regression testing of Codex components in MediaWiki (desktop and mobile) will be a significant asset for our work and will help us catch bugs in the short and long term. However, the current setup and workflow could use some optimization.
Pros:
- Established tool that may eventually be maintained by QTE
- Robust testing within MediaWiki
Cons:
- Workflow for testing Codex components is quite burdensome. Right now it requires pulling in a specific Codex commit locally in the VueTest repo, running code there to update the local version of Codex and copy over the new styles, then pushing a patch with those changes. Then, you can locally test that patch against reference images. This is not ideal for rapidly and frequently testing in-progress work.
- The Sandbox page is not completely optimized for easy VRT. We can target selectors, which means we can easily set up a test for each component's `<section>` element, but even that causes issues when there are slight changes (even just pixel rendering differences) that cause a vertical layout shift
---
### Add VRT to Codex
Summary: setting up a visual regression testing system in Codex would be lightweight and would allow us to quickly and frequently test both in-progress work and new releases. However, testing Codex components in isolation will only get us so much, and we will need to test them in MediaWiki to cover our most common and visible use cases.
Pros:
- We can create a lightweight system that can run locally, as part of CI, and when we do new releases
- No dependencies on other codebases (e.g. VueTest, Pixel) or MediaWiki
Cons:
- Need to set it all up ourselves
- Doesn't cover testing in MediaWiki
I think we should either do a simple BackstopJS setup (which is what Pixel uses) or consider a Cypress plugin if we think we'll use Cypress for other forms of testing. Another options we should explore is Microsoft's Playwright.
---
So, to do:
- Soon:
- Submit a patch to Pixel to add Codex components and discuss with the Web team
- Set up VRT in Codex
- Later:
- Improve the Sandbox page so we can avoid muddying the diff when there are vertical layout or size changes
- Improve the workflow for updating Codex code within the Sandbox