Overview

Reading web is working on displaying page issues more prominently on mobile web T159262. As part of our design process we will be conducing user research to explore design solutions and better understand how page issues affect the reading experience.

User testing outline
https://docs.google.com/document/d/137Zr6syzvaR2ntfx6C3TnMlc1ZT7QU5ig1qzaavZiaU/edit?usp=sharing

Goals of the study

1. To build empathy and understanding around how page issues affect the perception of article quality and reliability among readers
2. To build understanding of users’ level of familiarity with page issues and their perceptions around page issues of different severity levels
3. To get feedback on the visual treatment of how we display page issues in articles, specifically: the inclusion/exclusion of semantic titles and timestamps, and additional confidence that the new treatment is more noticeable than the previous one

Test details

This will be an unmoderated remote task-based test, conducted using UserTesting.com to recruit participants, and record sessions.

Variants
There will be 4 variants of the test:

1. Group 1 will see articles without page issues (control)
2. Group 2 will see articles with the current page issues treatment
3. Group 3 will see articles with the new proposed page issues treatment, style A
4. Group 4 will see articles with the new proposed page issues treatment, style B (titles+date)

There will be 4 participants in each group, spread out around the world.
1 older female, 1 younger female, 1 older male, 1 younger male

Participants will be screened for the following critieria:

• must be familiar with Wikipedia
• cannot be a Wikipedia super-user

Results

Top line results:
✓ The new treatment increases awareness of page issues among participants (particularly when they are in a more evaluative/critical mode)
✓ Page issues make sense to readers and they understand how they work
✓ Readers care about page issues and consider them important
✓ Positive sentiments towards Wikipedia associated with readers learning about page issues

The full version of the research results are here