Page MenuHomePhabricator

User testing of new onboarding and share-a-fact flows
Closed, ResolvedPublic

Description

Testing script

https://docs.google.com/a/wikimedia.org/document/d/1A314wMCK73kP9P2up61d_YN8xXb9L0ALkwEdOzmpwm8/edit?usp=sharing

Testing goals

In 5.7.0 we've updated a number of features to the app, which we would like to test for discoverability and usability, as well as collect qualitative feedback on the designs. The onboarding screens were updated based on findings from previous user tests and comparative analysis. The ‘Share-a-fact’ flow and card designs were updated to utilize more native components and to match the look and feel of the app. Additionally, swipe gestures were updated and added to new list areas in the app.

The primary goals of this test are to gain insights on the following aspects of the app:

  • Do first time users have a good understanding of the app and the apps features after onboarding and their first Explore feed load?
  • Are swipe actions easy for users to discover and utilize?
  • Can users successfully complete the ‘Share-a-fact’ flow?
  • Gather qualitative feedback on the new ‘Share-a-fact’ card design.

Test format

This test will utilize UserTesting.com to recruit users, record sessions and ‘facilitate’ the test.
The test is an unmoderated, task based test. Users will be given a set of tasks to complete in a 10-15 minute time frame, followed by a set of 4 written questions. Users will perform the test remotely.


Findings

Do first time users have a good understanding of the app and the apps features after onboarding and their first Explore feed load?
All but one user was able to name prominent features from the app after onboarding. The one user who did not know the apps capabilities did not read any of the onboarding copy that was over 2 lines. Generally all users commented on the easy to use layout of the Explore feed. One user was upset by the featured article of the day and two users wanted to have more customization control over the feed (one even before his first feed load).

Are swipe actions easy for users to discover and utilize?
4 of 6 users discovered and utilized the swipe actions. All users who used the swipe actions found them easy to use. One user did not like the trash icon for removing a saved article.

Can users successfully complete the ‘Share-a-fact’ flow?
All users were able to trigger the 'Share-a-fact' flow. iPad users were unable to share as I made a mistake in my test write up for iPads. The flow was not surprising for any users. Generally users found the share flow to be 'familiar.'

Gather qualitative feedback on the new ‘Share-a-fact’ card design.
A number of users could not see the Share-a-fact card preview (esp. on the iPhone 5) Generally users seemed to find the card to be a good 'preview' to an article.

Event Timeline

Moving into blocked or waiting as the testing script is ready, but I'll need an IPA for testing. Thanks!

  • Feed card too tall on shorter devices (5 and below)

https://phabricator.wikimedia.org/T178119

  • Share-a-fact share sheet icon is too large on iPhones

https://phabricator.wikimedia.org/T178105

  • Highlight menu does not always appear

Discussed with Joe, only reproducible 50% of the time on iOS 9

  • On this day looks a bit off in landscape on iPad

https://phabricator.wikimedia.org/T178121

  • Can we pause on the share-a-fact card on the share sheet

https://phabricator.wikimedia.org/T178122

  • Make the share a fact background overlay dark to help the card stand out more

https://phabricator.wikimedia.org/T178124

  • Make the large font size larger for share-a-fact cards

https://phabricator.wikimedia.org/T178123