Page MenuHomePhabricator

[Spike] Automated accessibility testing for Codex
Open, MediumPublicSpike

Description

With WVUI and it's Storybook test site, there was an accessibility addon available that uses axe to provide accessibility testing for each story. See their docs for details.

As we've moved away from Storybook in Codex per T290912, we should prioritize replacing this critical functionality, whether it's building something similar for our demos in VitePress, or integrating an automated testing workflow into our processes elsewhere.

Acceptance Criteria:

  • Audit of existing components for accessibility using out-of-the-box tool
  • For each component's audit, create individual tasks that need addressing
  • Modifying acceptance criteria for future components

Event Timeline

So looking into what the Storybook addon uses, it appears to be the Axe library at https://github.com/dequelabs/axe-core

It should be possible to use that with the wrappers that we get from mount() in the test cases. Untested, but I think something like adding an accessibility check to the snapshot cases might be enough. Example case with CdxButton

test.each( cases )( 'Case %# %s: (%p) => HTML', ( _, props, slot, done ) => {
	const wrapper = mount( CdxButton, { props: props, slots: { default: slot } } );
	expect( wrapper.element ).toMatchSnapshot();
	axe.a11yCheck(
		wrapper.element,
		{},
		function ( result ) {
			expect( result.violations.length ).toBe( 0 );
			done();
		}
	);
} );

aXe core would be our first choice. As Lucy Greco has shared from one of her a11y testing evaluations for UC Berkeley already 3(?) years ago at a Accessibility Camp Bay Area, aXe was the only testing tool not only to have least false positive results, it was also the only accessible tool in their evaluation.

Volker_E renamed this task from Consider automated accessibility testing in the new Vue component library to Consider automated accessibility testing for Codex.Apr 25 2022, 4:38 AM
Volker_E updated the task description. (Show Details)
Volker_E added a project: Design-System-Team.
STH renamed this task from Consider automated accessibility testing for Codex to Spike: Automated accessibility testing for Codex.Apr 25 2022, 12:35 PM
STH triaged this task as Low priority.
STH updated the task description. (Show Details)
ldelench_wmf updated the task description. (Show Details)
ldelench_wmf moved this task from Inbox to Needs Refinement on the Design-System-Team board.

I did some investigation from the links shared by Volker and the implementation created by Danny.

This are just for information only as more investigation and discussions are required, overall seems like one or more of the following options would be best for our workflow:

  1. Add a eslint plugin to ensure accessibility is taken care of while developing (this has been done by the Lexeme team T307912
  2. It is easy to implement
  3. It will ensure accessibility is done during development saving time when opening Patches
  4. it will break the build is it does not pass
  5. It can slow machine and development as most dev environment are set to run eslint on save!
  6. Add jest accessiblity tests on ALL of our test cases, or just one test per file. (this has been completed by the query builder team).
  7. Simple to implement (see screenshot below)
  8. It may result in many test cases to fail due to inconsistent or incomplete properties
  9. It may not be very useful if we already pass have the eslint plugin
  10. Create a Accessibility report in each documentation pages (not done anywhere, cannot find it on internet)
  11. We could create a table with the result of AXA API and show all the pass and failure of the component
  12. It would be great for component development too (as we use the docs for development too)
  13. It will provide a nice visual indicator to people using our component of their standards
  14. It may need a one off development effort

image.png (833×1 px, 134 KB)

Volker_E renamed this task from Spike: Automated accessibility testing for Codex to [Spike] Automated accessibility testing for Codex.Aug 15 2023, 4:29 PM
Volker_E added a project: Spike.
Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptAug 15 2023, 4:29 PM