Testing script

https://docs.google.com/document/d/135f-yBJJnHoyCdP-4PuLZMBjCpLupuMtdDmekEefE6s/edit?usp=sharing

---

Testing goals

Reading lists is a new feature on the iOS app. Along with releasing Reading lists to the beta testing group and testing internally, we will be utilizing an unmoderated, task-based user test to test the discoverability and usability of the feature as well as to receive qualitative feedback on the feature.
Reading lists are an iteration on the iOS app’s Save for later feature. Reading lists expand on this feature by allowing users to sort their saved articles into searchable lists.

Messaging

• Is messaging (and timing of messaging) related to the benefits of logging in / creating an account to enable syncing clear and easy for users to understand?
• Are empty states, warnings and search titles/suggestions clear to users and easy to understand and act upon?

Article saving and management

• Can users successfully create reading lists and add articles to them?
• Can users successfully locate and utilize the bulk editing view to manage their individual saved articles and reading lists?
• Do users understand how deleting a reading list affects their saved articles?
• Is the concept of the ‘Bookmarks’ (default list) easy for users to understand and use?
• Is the CTA to add recently saved articles to a list visible to users and easy to utilize?
• Is it easy for users to tell when new updates are available for their articles and reading lists?

Navigation + search

• Can users successfully navigate between the All saved articles view, the reading lists view and individual reading lists?
• Can users successfully locate the search on the ‘All saved articles’ view?

Testing Information

Test format

This test will utilize UserTesting.com to recruit users, record sessions and ‘facilitate’ the test.
The test is an unmoderated, task based test. Users will be given a set of tasks to complete in a 15 minute time frame, followed by a set of 4 written questions. Users will perform the test remotely.

Testers

5 users will be recruited using UserTesting.com.

---

Findings

From iOS team walk through

• HR between cell items should be updated to extend the full length of the cell (design bug, sorry!) https://phabricator.wikimedia.org/T186074
• Hints disappeared too quickly, can we extend by 3-5 seconds? https://phabricator.wikimedia.org/T186078
• Underline on the two views on the Saved tab should float like the underline on search and have 5px of L/R padding https://phabricator.wikimedia.org/T186074
• Consider adding long press actions to list items in reading lists https://phabricator.wikimedia.org/T186138
• Support filter, search and sort within a reading list https://phabricator.wikimedia.org/T186082
• Clear option should be added to the Saved articles view on the upper left corner https://phabricator.wikimedia.org/T186074

Usertesting.com

• User expect to articles to not be deleted when deleting a list (x 2), will need to test if this is clearer without the Saved list
• User thinks that you have to tap on the hint to save the article, we should update the hint text to be clearer https://phabricator.wikimedia.org/T186078
• Text in Reading list empty state needs to be darker https://phabricator.wikimedia.org/T186074
• User wanted an option to move articles from one list to another (not save additionally), consider if articles can be 'moved to' a new list rather than 'added to' a new list
• User wants to save articles into a reading list from the reading list view (not sure about this one)
• User wants to be able to sort a saved article into a reading list from the article view (eg. without seeing the hint, un-saving and saving), consider an action under the magic share menu, that is 'save to a reading list' or force press on save opening the add to list view) https://phabricator.wikimedia.org/T186086

Bugs and things to be implemented from original designs

• Un-saving an article removes it from the 'Saved articles' list but not from user created lists https://phabricator.wikimedia.org/T186083
• Articles that are saved into multiple reading lists, the reading list tags are not in a consistent order (eg. it might be 'a, b' on one list item and 'b, a' on another) https://phabricator.wikimedia.org/T186084
• Messaging not appearing after a user deletes a list by swiping on the list
• Header on reading list not yet implemented
• User education and not yet implemented
• Hints stay open indefinitely when scrolling is occurring https://phabricator.wikimedia.org/T186078

Second round of Usertesting.com

• 4 out of 5 users were able to complete all tasks quickly. One user had trouble finding the bulk edit view
• 5 out of 5 users expected that the default list was ‘unsorted’ eg. any articles that did not appear in other lists. One user was particularly upset that this wasn’t the case. No users stated that they found utility in this list, most wanted to be able to have it appear below their user created lists
• 5 out of 5 users understood the difference between the ‘all articles’ and ‘reading lists’ views. Users were able to easily sort their articles and one user was very excited about search.
• 5 out of 5 users spotted and used the hint to create reading lists (3 out of 5 created a new list unprompted after saving their first article)
• 5 out of 5 users expected their articles to not be deleted when they deleted a list. This behavior should be updated. https://phabricator.wikimedia.org/T187602
• Truncation on the hints was a big problem for two users https://phabricator.wikimedia.org/T187281