Adding automated accessibility testing to our CI tests will allow us to quantify a11y issues over time. When used with manual testing, automated testing can help catch new issues, and be used to evaluate the impact of a11y improvements to Skins.
Create automated accessibility tests using [[ https://github.com/dequelabs/axe-core | axe-core ]]. The tests will focus on common Skin UI rather than article content or user generated content, which would generate a lot of noise. The tests will also not cause CI to fail when violations are found, but will instead record data from the tests to be used in Grafana. Automated a11y tests will be run daily along with the other daily selenium tests.
Ideally as much of the accessibility testing boilerplate and config can be in core, so that setting up tests in different Skins is as easy and consistent as possible.
### Proof of concept
Patch in Vector: https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/771476
Currently the POC only sends a very simple count of a11y violations & warnings to Graphite via the statsv endpoint. In the future we should be able to log more details to be viewed in Grafana. We also should migrate to Prometheus as Graphite is deprecated