Make visual regression tests run in CI (non-blocking) for the Vector repo
Closed, DuplicatePublic
Actions

Assigned To

None

Authored By

	• nray
	May 11 2022, 10:05 PM

Description

The easiest way to use visual regression tests is to have a computer run them for you automatically. This task proposes adding a non-blocking CI job to the Vector repo that runs on each patch set made to Vector. If that goes well, other repos could be added.

It would run something like the following which would compare master against the change on top of master:

./pixel.js reference && ./pixel.js -c <Change-Id of Gerrit Patch>

The preceding command launches several Docker containers (e.g. visual regression test container, a container with MediaWiki, a container with the database) and runs the visual regression tests. If that command exits with 0, the job passes. If it exits with something else, it fails. However, failures shouldn't block merging and should only be seen as flags that need discretionary review.

Pixel already does a similar command for commits made to it.

Acceptance Criteria

For each Gerrit patch set made to Vector, CI runs the command above which takes a visual diff of master against the change/changes on top of master.
The job should provide a publicly accessible link to the report generated (e.g. see https://pixel.wmcloud.org/desktop/ for example) where one can review the screenshots taken.
If the change includes dependencies (Depends-On) that Pixel doesn't support, the job can be skipped.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• nray	T302246 [GOAL] Leverage Automated Visual Regression Testing
		Duplicate		None	T308194 Make visual regression tests run in CI (non-blocking) for the Vector repo

Event Timeline

• nray created this task.May 11 2022, 10:05 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 11 2022, 10:05 PM

• nray updated the task description. (Show Details)May 11 2022, 10:09 PM

• nray added a parent task: T302246: [GOAL] Leverage Automated Visual Regression Testing.May 11 2022, 10:12 PM

• nray updated the task description. (Show Details)

• nray added a project: Web-Team-Backlog (Needs Prioritization (Tech)).

• nray moved this task from Backlog to UI Regression Testing on the Web-Team-Backlog (Needs Prioritization (Tech)) board.May 11 2022, 10:14 PM

• nray updated the task description. (Show Details)May 11 2022, 10:26 PM

• nray updated the task description. (Show Details)

• nray updated the task description. (Show Details)May 11 2022, 10:36 PM

• nray updated the task description. (Show Details)May 12 2022, 8:57 PM

For each Gerrit patch set made to Vector, CI runs a command that takes a visual diff of master against that change/changes on top of master.

Although our current CI is terribly dated and legacy, we have all the logic to do that. The flow is roughly:

Zuul listens for Gerrit events and triggers jobs
The jobs are defined in jobs in Jenkins
A job run a Docker image which has all the execution environment (MariaDB, php, composer nodejs, npm etc)
The image runs Quibble the MediaWiki test runner https://doc.wikimedia.org/quibble/ which holds the logic to run things

It is easy for us to add a new job which would npm run pixel for example so most of the logic would be delegated to the repo / the npm package.

We have a few use cases doing comparison with the parent of the change being tested. There are different ad hoc implementations though. Usually that boils down to something like:

run the command against the patch saving the result to an artifacts directory
checkout the parent commit (HEAD@^1)
run the command
do the diff

Order might vary ;)

The job should provide a publicly accessible link to the report generated where one can review the screenshots taken.

Upon completion of the job, the CI system will report back to Gerrit with a link to the job :)

If the change includes dependencies that Pixel doesn't support, the job can be skipped.

Not quite sure about this one.

• nray updated the task description. (Show Details)May 13 2022, 12:32 AM

• nray updated the task description. (Show Details)May 13 2022, 12:35 AM

kostajh added a subscriber: kostajh.May 13 2022, 9:10 AM

• nray updated the task description. (Show Details)May 16 2022, 8:24 PM

Thank you for your response @hashar. I have a follow up question below (I'm also happy to meet with you if that's easier):

In T308194#7925957, @hashar wrote:

Although our current CI is terribly dated and legacy, we have all the logic to do that. The flow is roughly:

Zuul listens for Gerrit events and triggers jobs

The jobs are defined in jobs in Jenkins

A job run a Docker image which has all the execution environment (MariaDB, php, composer nodejs, npm etc)

The image runs Quibble the MediaWiki test runner https://doc.wikimedia.org/quibble/ which holds the logic to run things

It is easy for us to add a new job which would npm run pixel for example so most of the logic would be delegated to the repo / the npm package.

Our visual regression tool is currently a CLI that launches a set of Docker containers extended from MediaWiki-Docker (e.g. there is a Docker container to run Mediawiki with the LocalSettings.php we want, a Docker container for MariaDB with seed data that the tests use, etc. Is it possible to launch these Docker containers within our CI job? The process to use the tool would look something like this:

npm install -g pixel
pixel reference && pixel test --change <Change-Id of Gerrit Patch> --output <path-to-report> . These commands launch the Docker containers, takes screenshots of the page before and after the change, and outputs a static report directory that contains an index.html/screenshots of the results (e.g. see https://pixel.wmcloud.org/desktop/index.html). I was envisioning this report folder would be the artifact produced from the job.