Testing details

Testing script

https://docs.google.com/document/d/1ADqFi3hyzCqow_Qxgr0ZpVdhZYEUJMIQyG6T6lM_4dU/edit?usp=sharing

Testing goals

With 5.6.0 we've added a number of new features to the app, which we would like to test for discoverability and usability. Additionally there have been updates to the feed design and feed cards since our last round of user testing, so qualitative feedback on new card designs is also desired. Finally, updates to the app on-boarding are anticipated in a near future release, so testing the onboarding flow is also beneficial.

The primary goals of this test are to gain insights on the following aspects of the app:

• Do first time users have a good understanding of the app and the apps features after onboarding and their first Explore feed load?
• Can users successfully find and use the appearance settings / reading themes
• Can users successfully interact with elements on the new ‘On this day’ card?

Features to be tested

• Onboarding
  • Do the onboarding screens make sense to users?
  • Do users understand why they are being asked
  • Is anything missing?
  • After completing the onboarding screens, what are user's thoughts on how onboarding prepared them for using the app?
  • General quantitative feedback on onboarding
• Reading themes
  • Feed card (Do users notice it? if so do they understand it?)
  • Are users able to modify their reading theme? (either in settings or in the article view)
  • Is the disabled state of Dim images on default and sepia clear to users?
  • Do users find the ability to dim images useful? Preference: are images dimmed too much, too little?
  • Is anything missing? Do users wish they could do something else with the feature?
  • General quantitative feedback on reading themes
• On this day
  • Can users successfully navigate to the detail view?
  • Does the date ordering make sense to users?
  • Can users successfully use the horizontal cards?
  • Is anything missing? Do users wish they could do something else with the feature?

Testing Information

Test format
This test will utilize UserTesting.com to recruit users, record sessions and ‘facilitate’ the test.
The test is an unmoderated, task based test. Users will be given a set of tasks to complete in a 15 minute time frame, followed by a set of 4 written questions. Users will perform the test remotely.

Testers
5 users will be recruited using UserTesting.com.

---

Findings

Overview

7 users in total went through the flow, with 5 completed tests (one had audio issues, the other did not read the questions). Generally users we excited about the Wikipedia app! There were a number of very active readers of Wikipedia and even a user who edits Wikipedia. These 'power readers' really liked the customization available to them in the app (recommended reading and appearance preferences) additionally these users were often excited about the TOC and search suggestions.

Primary findings requiring new tickets

• All users are still having trouble understanding the dim images toggle on default and sepia. It's clear to most users that the toggle is inactive, but they don't understand why https://phabricator.wikimedia.org/T172010
• Users primarily associate the tT icon in the article tool bar with text size, although it was clear to users once they clicked why reading themes would be included under this icon. It might help to make the 'tT' illustrations stronger on the feed card. https://phabricator.wikimedia.org/T172014
• Horizontal cards on On this day look like single cards on smaller devices (eg. there is no second article 'peeking' for smaller screens) https://phabricator.wikimedia.org/T172017
• Keyboard on 'Create account flow in dark mode is transparent https://phabricator.wikimedia.org/T171971
• Circle on appearance popover for dark mode uses the wrong dark color https://phabricator.wikimedia.org/T172011
• The feed appears to be jumping a bit when the notification card is added into the feed https://phabricator.wikimedia.org/T172019

Additional findings

Language

• The majority of tested users could not identify the Language icon in the article toolbar
• All users tapped on 'English' in the onboarding language screen

Onboarding

• Two active users of Wikipedia mentioned that they would have liked to have more information in onboarding around how the app is different from web
• One user mentioned that he would have an option for a small 'tour' of the app
• The majority of users liked that the onboarding process was short (no login, etc)

Explore feed

• Users were divided on infinite scroll and it was often hard for users to tell initially that the feed is in backwards chronological order
• Generally users found the explore feed to be engaging, the majority of users mentioned the large images as appealing
• The majority of users mentioned that they understood why the Top Read articles were appearing in Top Read and liked the feature.
• One user did have trouble understanding the Top Read card and thought it was editors whose articles were top read on Wikipedia
• One user wished that she could hide the tab bar on the Explore screen

On this day

• Generally users responded positively to the On this day card
• Two users were confused by which event was selected for the card, although one guessed that we were trying to not show tragedies as the featured card
• One user was very, very excited by the dot animation!
• For users with smaller screens, it was unclear if they knew that the horizontal cards could be swiped to reveal more cards
• Users seem generally familiar with the small card presentation, one user mentioned that it reminded her of link sharing on iMessage
• Some general comments: It would be nice to share this with students as trivia, it brings back memories.

Reading themes + Article view

• Users generally had a positive response to the theme colors
• Once they were able to activate it, users generally had a positive response to image dimming
• One user requested the ability to switch between serif and san serif
• Two users would have liked an option to remove images from articles
• Two active Wikipedia readers really enjoyed having the TOC although both initially ID'd the tool bar icon as 'menu'
• There is an increase usage of the W-button with the addition of the popover! Also a two users mentioned liking having the W-button as a quick way to return home
• One user experienced some horizontal scrolling in quick facts (he was on a smaller device), which he noted as 'sloppy'
• One user on a smaller device wished that the appearance popover was smaller

Search

• Two users recovered their mis-tap or misspelling in search by returning to a previous search term.
• Generally users liked having search suggestions
• The majority of users utilized the exposed search at the top of the Explore feed to perform their first search

Places

• Across tests, we've now had a few users who have thought that the Places tab shows what people are searching for by location.

Notifications

• In onboarding one user was put-off by the first screen as she is weary of having more notifications on her phone. Unclear, but it seems like she might have thought she couldn't opt out as she wasn't given the opportunity to opt in.
• One user expected to have a notification option for 'article of the day'