I talked to each dev on the team and asked them about the biggest pain points they were having with Pixel. The following themes emerged:

• 2 people on the team felt the lack of a process around the intentionality of changes (e.g when you view https://pixel.wmcloud.org/reports/desktop/index.html, see lots of failures, and don't know who or what caused them and whether they were intentional) was the biggest pain point

• 2 people on the team felt that needing to run Pixel locally to compare a patch with master was the biggest pain point. Instead, it should run in CI.

• 1 person felt that the length of time Pixel takes to run tests was the biggest pain point. They would like the ability to run a single test instead of an entire test group.

All of these criticisms are valid, but I personally feel that the lack of context (intentionality and attribution) around changes that appear on https://pixel.wmcloud.org is the biggest problem that is most lacking a viable solution right now. Currently, if you want to know the context around the failures that appear on https://pixel.wmcloud.org, you need to manually hunt them down yourself by either going through commit messages, Phabricator tasks, or asking people. It can be a pretty big time waster.

One idea to help with this issue is to run Pixel on every merged commit in the master branches. If a report has failures, it would be displayed in a list of failed reports for that test group that you could select and view. Additionally, it would display the commit message of the commit that led to the failures so that attribution and context is much better understood. I drew up the wireframe below: