Pulling back into backlog since the task hasn't seen any action in >1 month; feel free to adjust.

Hi all, I believe these guidelines are ready for cross-team review. I've made all the updates I feel are necessary to the "testing in MediaWiki" part of the document (and unblocked the technical barriers that previously kept this from being possible).

Let's circulate this doc for appropriate cross-team review so that we can wrap up T306180: [Epic] Define manual testing & QA strategy in Codex .

I'm sure WMDE engineers would be happy to take a look at these, so feel free to list us under 'Teams to reach out to' if you think we could help :)

If folks want to see a practical example of this workflow (fixing a bug that only shows up on MW using VueTest and PatchDemo), this phab task provides a good use-case:

T324495: Dialog styles getting overwritten

Please find below my review. Overall, I think there are some really good ideas and resources in the document, particularly the different test ideas listed in step 3.

I have 5 main comments:

• Too much focus on the specification as the primary or only source of truth. Not enough focus on the risk that ultimately guides this focus, which I assume to be consistency between different Codex components.
• Not enough prominence given to other risks or guidance about test ideas which are not based on the spec.
• Too much faith in up-front planning rather than responding to new information.
• There are a couple of words and phrases I don't understand.
• There are a few small mistakes.

Specification

The Spec Sheet is the source of truth for visual/functional requirements of components.

What if the spec is wrong? Or incomplete? Or out-of-date? What about risks that are not mentioned in the spec? What if the spec if buggy with respect to another risk (e.g. accessibility)? What parts of the spec should I pay attention to and what parts should I ignore (we always are making these decisions, even if we are not aware of it).

What if I don't have a spec, can I not test? Testing might become dependent on design/PM, which might cause a bottleneck when valuable testing work could be done even without a spec.

Would it even matter if the component was inconsistent with the spec? Is the risk that if it is inconsistent with the spec it will be inconsistent with other Codex components? Is this the overriding concern of this project, and testing against the spec a means to this end? Perhaps that should be stated up-front so we don't confuse means with ends. Testers are likely to be able to do a better job if they understand the ultimate goal they are trying to achieve.

[If I may make a personal appeal, there is a risk here of treating the tester as a cog in a wheel who is fed specifications and outputs test cases, rather than as an autonomous member of the project.]

I guess we are assuming that all the specifications for Codex components are consistent with one another and that all testers are testing consistency against the spec in the same way. So, if the implementation is consistent with the spec then it is consistent with all other components, because "consistency" is considered a transitive relation. But this is assuming a lot and might not work in practice.

Other risks

It is arguable that all testing ultimately boils down to risk. "Inconsistency with the spec" is one such risk, and even the way we test for this is driven by other risks. For example, I notice when viewing Figma documents that if I scroll up and down really fast the text does not fly off the screen. Should I test that the implementation is consistent with this? Probably not, because this is not a risk based on my understanding of how browsers work. On the other hand, having the incorrect amount of padding is a plausible risk.

The document does list a few risks other than inconsistency with the spec, such as:

• "Global stylistic properties: padding, sizes, fonts, etc"
• "Making sure the right states and actions are displayed when the component is clicked, selected, searched, text input etc."
• "Responsiveness", "RTL behaviour", "accessibility"

but perhaps this could be expanded, given more prominence and testers encouraged to come up with their own ideas of risks.

The document also advises making a judgement based on the change being made. But, it does not have much advice about doing this. Again, a discussion of risks might be a good start for testers to decide what to focus their testing on.

For example, in step 3 we say:

The scope of the change...will determine the type of test or tests that would need to be performed...

Second bullet point of step 3:

For changes to existing components, execute the tests cases only in the relevant scenarios below (with regards the scope of the change)...

The second paragraph of step 3:

The following are the possible test approaches, to be performed based on their suitability depending on the task goals and acceptance criteria. Team members might feel free to decide the most convenient combination of tests and their scope, based on their expertise and available resources.

Planning up-front and responding to new information

Regarding step 2, whether or not a tester writes test cases in advance is a matter of personal preference and context. Some do, some don't. Some do for some things but not for others.

(As an aside, I don't have access to the Test Case document so I don't know exactly what the document means when it says "test cases".)

Moreover, whether you plan first and then execute is a matter of personal preference and context. For example, some testers follow a more "exploratory" approach where planning and execution happen simultaneously. In practice, testing (like many things) rarely happens in a linear fashion. It is more iterative/cyclical (although I appreciate it is hard to express something like this in a linear document).

The last paragraph of step 2 does state:

Test cases should both reflect the predefined component specs and properties, and attempt to explore how to break the component.

We should perhaps say this earlier because it might be easy to miss. I would also argue that test cases are not necessarily a good approach for exploring unknowns.

Things I don't understand

In step 2, it states:

Mapping the specification sections, typical test cases would cover the following

But I am not sure what this means.

One item in the bullet points in Step 2 is "use cases". What do we mean by these? Where do they come from? Are they meant to be the original reason the development team is creating the new component? Can they be tested via Special:VueTest? Would you have to ask the developers to create a realistic example?

Step 3 says:

In case of doubt, you can reach out to your friendly test engineer to help you define the best manual testing strategy.

Isn't this a bit recursive? Who is this document for? If you are a test engineer, do we assume you don't need this document?

Possible mistakes

Step 5 says:

Once signed-off, the manually tested ticket should be moved to the “Functional testing” column...

But the final point of "Manual testing checklist" says:

Once signed-off, a manually tested ticket should be moved to the “Product Sign-Off” column...

I assume this is a mistake.

Some of the links are to notion.so:

• Screen reader and keyboard navigation testing
• Color contrast
• find recommended screen reader software in the section below
• More about mobile testing browsers and breakpoints
• more about testing for accessibility