This task will be to re-run the same test protocol as written in T260534 with a single Concept A design, incorporating revisions based on recommendations from that study.
Goal of this user test
Evaluate to what extent the revised design addresses issues identified from the first round of testing.
Testing format
Due to time and personnel constraints, this will firstly be an unmoderated remote task-based test, conducted using UserTesting.com to recruit participants and record sessions in English only.
Testers
- The intention will be to screen for respondents who are new to editing on Wikipedia.
- Mobile users (as this is for the mobile version of the task)
- Android devices only – as the prototype unfortunately relies on sticky bottom panels that do not render properly oniOS devices.
Test protocol
https://docs.google.com/document/d/1znt3Px2PDN6FPth9mfr2mJ5Vd8hINKWmyVkvXqj65HQ/
Results summary
Key issues and areas of improvement from round 1 & how they performed in round 2
| Issue | Amendments in Round 2 | Round 2 results |
|---|---|---|
| Understanding of add links task before article | Updated copy to refer to Adding links between wikipedia articles | Testers seemed more likely to be correct about what adding links is. |
| Onboarding reflexively being dismissed by many participants | Onboarding appears as soon as users open the task for the first time | No one dismissed the onboarding reflectively. |
| Onboarding information not conveying how to evaluate links | * Copy changes * Splitting info to multiple screens * Providing illustrations * Providing examples * Guidelines as a standalone step | Onboarding was much better. Multiple users appreciated the example sentence with highlights. |
| People were focusing too much in general on the card/bottom sheet (e.g., wikidata description match to article) to evaluate links, not enough on article context | * Replaced wikidata description with article extract * Updating copy in the task description * Adding a help icon on the suggestion card allowing constant access of evaluation guidelines | No improvement. Users focused right on the word and did not read the surrounding sentence. |
| People were unsure if they could preview suggested articles | Addition of an ‘open in new window’ icon next to the article title. | Unknown, since no one mentioned wanting to investigate the article linked more. |
| “Skip” was not explicit | Adding a “skip” label before the “>” icon | Users were able to skip just fine. |
| Some people were not quite sure how to publish | Auto-advance users after they make a selection on a suggestion | Users are generally fine with auto-advance and it causes them to move quickly. However, one user wished that they could confirm before auto-advancing, but he didn’t say why. |
Full document: https://docs.google.com/document/d/1kyCzjjzq_7zwsoQPMP8MSiPwQOthjzxBev82b_lB2GE/