Testing script

https://docs.google.com/document/d/1ARmYAIBWJhX0mOSEKjBnQlTBLYGEskDyB840sxU79t8/edit?usp=sharing

Testing goals

Reading lists is a new feature on the iOS app. Along with releasing Reading lists to the beta testing group and testing internally, we will be utilizing an unmoderated, task-based user test to test the discoverability and usability of the feature as well as to receive qualitative feedback on the feature.
Reading lists are an iteration on the iOS app’s Save for later feature. Reading lists expand on this feature by allowing users to sort their saved articles into searchable lists.

Messaging

• Is messaging (and timing of messaging) related to the benefits of logging in / creating an account to enable syncing clear and easy for users to understand?
• Are empty states, warnings and search titles/suggestions clear to users and easy to understand and act upon?

Article saving and management

• Can users successfully create reading lists and add articles to them?
• Is the CTA to add recently saved articles to a list visible to users and easy to utilize?
• Do users understand the value of creating an account to sync their articles?
• Is the process of turning article sync on and off clear and easy to follow for users?

Navigation + search

• Can users successfully navigate between the All saved articles view, the reading lists view and individual reading lists?
• Can users successfully locate the search on the ‘All saved articles’ view?

Testing Information

Test format

This test will utilize UserTesting.com to recruit users, record sessions and ‘facilitate’ the test.
The test is an unmoderated, task based test. Users will be given a set of tasks to complete in a 15 minute time frame, followed by a set of 4 written questions. Users will perform the test remotely.

Testers

5 users will be recruited using UserTesting.com.

Findings

• T189169 5 out of 5 users were shown the 'turn syncing on' popover after logging into an account that already has sync turned on. This caused a lot of confusion for 3 out of 5 users.
• T191076 All users thought that they could log in on desktop and see their synced articles
• T191096 The majority of users had trouble figuring out where to turn sync on after dismissing initial pop-ups / the feed card.
• T191040 1 user had the articles in her user created reading list duplicated (eg. red panda showed up twice in a single list)
• T191073 1 user had a hard time understanding the empty state text on the Create a new list screen. We should update the string to use 'reading list' vs. 'list'
• 2 users had trouble finding the save button on the article view, although this has not been a problem on other tests and the design has not changed