Page MenuHomePhabricator

Add automated accessibility tests in CI to generate accessibility benchmarks for Skins
Closed, ResolvedPublic

Description

Background

Adding automated accessibility testing to our CI tests will allow us to quantify a11y issues over time. When used with manual testing, automated testing can help catch new issues, and be used to evaluate the impact of a11y improvements to Skins.

Proposal

Create automated accessibility tests using axe-core. The tests will focus on common Skin UI rather than article content or user generated content, which would generate a lot of noise. The tests will also not cause CI to fail when violations are found, but will instead record data from the tests to be used in Grafana. Automated a11y tests will be run daily along with the other daily selenium tests.

Ideally as much of the accessibility testing boilerplate and config can be in core, so that setting up tests in different Skins is as easy and consistent as possible.

Proof of concept

Patch in Vector: https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/771476
Grafana: https://grafana-rw.wikimedia.org/d/9ejT-XA7k/readers-web-a11y?orgId=1&from=now-7d&to=now

Currently the POC only sends a very simple count of a11y violations & warnings to Graphite via the statsv endpoint. In the future we should be able to log more details to be viewed in Grafana. We also should migrate to Prometheus as Graphite is deprecated

Event Timeline

Change 740684 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[mediawiki/skins/MinervaNeue@master] POC Add automated a11y tests to selenium with axe-core

https://gerrit.wikimedia.org/r/740684

Change 740684 abandoned by Bernard Wang:

[mediawiki/skins/MinervaNeue@master] POC Add automated a11y tests to selenium with axe-core

Reason:

Abandoning in favor of pa11y approach
https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/771476

https://gerrit.wikimedia.org/r/740684

Change 771476 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[mediawiki/skins/Vector@master] POC Add automated a11y tests with pa11y

https://gerrit.wikimedia.org/r/771476

Change 771476 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Add automated a11y tests with pa11y

https://gerrit.wikimedia.org/r/771476

Jdlrobson added a subscriber: Jdlrobson.

This is currently being worked on so moving into board for tracking.

Jdlrobson triaged this task as Medium priority.Apr 6 2022, 9:44 PM

Change 777884 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[integration/config@master] Add selenium daily job to Vector

https://gerrit.wikimedia.org/r/777884

The selenium-daily-beta-Vector job now sets WMF_JENKINS_BEACON_URL=https://meta.wikimedia.org/beacon/statsv/?

Change 777884 merged by jenkins-bot:

[integration/config@master] Add selenium daily job to Vector

https://gerrit.wikimedia.org/r/777884

The tests are running well inside the new selenium-daily job for Vector, and results are showing in grafana. However, the reports are still not showing up in Jenkins, I'm still figuring out why the log artifacts arent showing.

From the job terminal output (selenium-daily-beta-Vector/31):

Archiving artifacts
No artifacts found that match the file pattern "log/". Configuration error?

From the job template:

    publishers:
...
     - junit:
         results: 'log/junit*.xml,log/WDIO.xunit*.xml'

Looks like the job is only archiving some xml files.

Looks like archive-log-allow-empty publisher is actually archiving the log folder.

job-templates.yaml

publishers:
 - archive-log-allow-empty

macro.yaml

- publisher:
    name: archive-log-allow-empty
    publishers:
     - archive:
        artifacts: 'log/'
        allow-empty: true

Change 789186 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/skins/Vector@master] a11y: create report in log folder

https://gerrit.wikimedia.org/r/789186

Change 789186 merged by jenkins-bot:

[mediawiki/skins/Vector@master] a11y: create report in log folder

https://gerrit.wikimedia.org/r/789186

Change 789643 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/skins/Vector@master] a11y: create report in LOG_DIR or 'tests/a11y/log'

https://gerrit.wikimedia.org/r/789643

Change 789643 merged by jenkins-bot:

[mediawiki/skins/Vector@master] a11y: create report in LOG_DIR or 'a11y/'

https://gerrit.wikimedia.org/r/789643

nray reassigned this task from nray to bwang.
nray added a subscriber: nray.

Great job @bwang !