Goal definition
Specific: What do we want to achieve?
Our visual regression tool (Pixel) is more stable and has less tech debt as defined by the criteria below.
Measurable: How will we know when we've reached our goal?
The following tasks and objectives have been completed preferably before I leave for sabbatical on July 5:
-
Pixel is versioned (T307936 in progress)(See T307936#8020407) - The database can easily be reset on each test run by passing a flag (e.g. --reset-db) (Planning to include this in T307936 in progress). This should make it easier to add tests that change the state of the database.
- Add visual regression tests for the typeahead search component T306846 . The typeahead search component is one of the most frequently occurring visual regressions and the tests will also be helpful when the wvui implementation is replaced with the codex implementation.
- Revise/fix visual regression checkout logic and increase unit test code coverage T309739
- Pixel's report on https://pixel.wmcloud.org/ and cron job should survive the Wikimedia Cloud VPS server restarting. Currently, I believe manual intervention is required to start es-dev-server and screen as described in T305563#7879426 . Can we point the apache server that is already setup at the report's static directory and remove es-dev-server?
- Also, the Cloud VPS instance volume has limited space and /srv/pixel.log file currently grows without limits. Can we install logrotate?
- The regression tests are consistently deterministic. We are getting close to this, but I've noticed one or two of the tests (e.g. MediaWiki_Test_sticky_header_vector-2022_logged-in_scroll_0_viewport_2_desktop) are sometimes flaky. Running all tests (desktop, mobile, echo, and forthcoming search tests) 10 times against the same code should consistently pass. See T309742#8045516
- The CI job that currently only runs the desktop tests for each PR should be changed to run all tests and should block merging into the main branch if they fail. See https://github.com/wikimedia/pixel/pull/85
- Github/npm ownership is transferred from nicholasray to wikimedia (T306731). If this proves too difficult or time consuming, at least ensure that each member of the team has the access they need to the repos.
Achievable: What support will we need to achieve our goal?
Support from other teams is most likely not needed except maybe for the last step (I'm not exactly sure what goes into the wikimedia github ownership process). Apart from that, dedicated time and space will be required to complete the above tasks within a month.
Longer term outlook
Although we won't have time to experimentally add Pixel to Vector's CI pipeline T308194 this month, completing the above tasks and improving the stability of Pixel will help facilitate that work.