Page MenuHomePhabricator

Perform user testing of Places feature
Closed, ResolvedPublic

Description

Testing doc: https://docs.google.com/a/wikimedia.org/document/d/1ZeDZ2BxQvf6Yuf4G-0mLIvbNS3oS8bsGlJYqIpUH72I/edit?usp=sharing

Places is a new feature on the iOS app. Along with releasing Places to the beta testing group and testing internally, we will be utilizing an unmoderated, task-based user test to test the usability of the feature as well as to receive qualitative feedback on using the feature in a variety of geographic locations. The test will be performed using UserTesting.com

The primary goals of this test are to gain insights on the following aspects of the Places feature:

Use cases
Interesting articles near current location

  • Can users locate interesting articles near their current location?
  • Can users navigate to their current location?
  • Can users successfully pan around the map, zoom in and out and perform searches in subsequent locations?
  • What is the (user-reported) quality of top articles?

Plan for a trip/research

  • Can users successfully locate articles of interest to them in a non-local location?
  • Can users successfully save these articles for later?
  • Can users successfully view their saved items on the map?
  • Can users successfully edit their saved list (either on Saved tab or Places tab)?

Messaging

  • Is messaging (and timing of messaging) related to the benefits of turning on location services (during onboarding or on first load of the Places feature) clear and easy for users to understand?
  • Are the icons used throughout the feature (map and list icons in navigation, my location button, tab bar icon) easy for users to recognize and understand?
  • Are empty states, warnings and search titles/suggestions clear to users and easy to understand and act upon?

Navigation + search

  • Can users successfully navigate between the map and list views?
  • Can users successfully save, share and read an article from the map view?
  • Can users successfully locate and use the ‘my location’ button on the map view?
  • Can users successfully navigate to an article from the list view?
  • Can users successfully locate and utilize the search bar (including clear their search)?
  • Are clusters and pins on the map view easy for users to understand and use?

Related Objects

StatusAssignedTask
ResolvedJoeWalsh
ResolvedJMinor
Resolved AMroczkowski
ResolvedJoeWalsh
ResolvedJoeWalsh
Resolved AMroczkowski
Invalid AMroczkowski
Declined AMroczkowski
Resolved AMroczkowski
Resolvedcmadeo
Resolved AMroczkowski
Invalid AMroczkowski
OpenNone
Resolved AMroczkowski
ResolvedJoeWalsh
ResolvedJoeWalsh
OpenNone
ResolvedJoeWalsh
ResolvedJoeWalsh

Event Timeline

cmadeo created this task.Feb 16 2017, 7:52 PM
cmadeo updated the task description. (Show Details)Feb 21 2017, 11:04 PM
cmadeo updated the task description. (Show Details)Mar 14 2017, 9:47 PM

Initial findings:

  • Thumbnails should be shown at further zoom level on iPad (hard to tell what the level should be without the slider)
  • Some users have a hard time recovering from filter searches. We might want to rethink how users can return to the default search of 'top articles' or clear their search.
  • It's very easy to accidentally select the filter search and receive no results. We might want to re-order which is the top result in searches and when the search button is enabled on the keyboard.
  • Search might need an empty state. Users often clicked on the search bar only to close it.
  • 4 out of 5 users panned to new areas to explore rather than searching.
  • 3 users wanted a more integrated map and list view.
  • Users did not seem confused by the interaction of clicking on a cluster, but it took users a fair amount of time to understand what the clusters were and what they represented.
  • Most users wanted to have new searches automatically triggered when they zoomed in or out.
  • No users discovered 'view saved articles on a map' all went to the saved tab instead.
  • Generally users wanted more information about the app in the onboarding screens (including some branding) 3 users also requested tutorials within the Places tab or per feature/tab.
cmadeo updated the task description. (Show Details)Mar 14 2017, 11:07 PM
cmadeo added a comment.EditedMar 16 2017, 7:17 PM

Create ticket:

  • Thumbnail logic based on density not zoom
  • Have the logic change for clearing search
  • Move filtered search to second on list
  • If entered before load then wait to submit

Add to tickets:

  • Empty empty state for search
  • Add parallel testing to ticket

Going to put off writing up a full report until user testing is performed on the newer version of Places

cmadeo added a comment.EditedApr 25 2017, 5:51 PM

Findings:

  • Increased usage of search bar since last test
  • Users are more successful in completing searches for geographic locations
  • Users are able to quickly locate the 'center me' button
  • Users are able to quickly locate the toggle between map and list
  • There didn't seem to be much of an issue for users understanding the cluster interactions. Many users assumed clicking on the cluster would zoom them in (there was some confusion when the number and the number of articles in the zoom-in did not match).

Create tasks for suggested improvements:

  • Increase tap area for search filters
  • Centering on location should clear previous search
  • Tapping on the 'x' to clear a search should just clear the search string without popping up the search overlay
  • Reconsider placement, color and text of the 'search this area' button. Perhaps keep it in the same place but color it blue and change the text to 'view results in this area' it's not intuitive to users that by browsing they are performing a search
  • Although searching for geographic locations has been greatly improved users are still having trouble locating specific articles on the map through search (eg. not waiting for the suggested searches to load). We might want to disable filter searches or pop-up a 'did you mean' suggestion after they press enter?
  • There are some issues with the search string and icon overflowing on some devices (one iPhone 7 and an iPhone 5c)
cmadeo added a comment.EditedApr 25 2017, 6:43 PM

Tasks to create:

cmadeo added a comment.EditedApr 26 2017, 12:02 AM

Moved to Blocked or Waiting until the following findings are addressed:

Then I'll test again!

cmadeo added a comment.EditedJun 1 2017, 6:28 PM

Findings

  • Overall the search experience appears to be greatly improved for users:
    • One user had an issue locating 'London' although the correct search result was the first result. He easily resolved this issue.
    • One user was confused by the string "Nearby matching 'search string'" although it should be noted that his connection was slow. We might want to consider removing this string until after other results have loaded.
    • 'Did you mean' helped to recover a user's search
    • One user had a hard time figuring out what types of things she could search for (eg. she searched for 'fireworks'), it might be helpful to update the default search box string
  • User continued to easily locate and identify the list / map toggle and search bar
  • One user had trouble locating the 'center on location' button, but was able to recover from this relatively quickly
  • All users correctly identified article clusters and either expected the clusters to zoom them in or were not bothered by the animation
  • Two users did not seem to notice the 'Results in this area,' this was hindered the search of one user in particular (she searched for Chicago and only two results showed on the map). The other user did not mention an expectation that he would see more articles.
  • 4/5 users really liked the compass animation on the list view
  • 3/5 users requested a way to save and share from the list view
  • Visibility of search filters dropdown is still low, but was used by one user and opened by another user.
  • Users appeared to have an increased awareness of why a limited number of articles is shown on the map (eg. the 'Top read' seemed to be more clear)
  • One user was especially bothered by the quality of the images on the list view

Suggested updates (tickets to be created after iOS task sync):

Some list items appear to have extended summaries on the list view (view clip above for an example)

This is fixed here https://github.com/wikimedia/wikipedia-ios/pull/1490/commits/ee8f00d3f3bf49f7232555dd81fe112834174db1

Occasionally the arrow indicator for 'Read' on the article popover is flipped

https://phabricator.wikimedia.org/T166773

cmadeo added a comment.Jun 1 2017, 9:52 PM

I've created tickets for the most recent findings. Although it would be great to do some more formal research on Places usage in the future for further feature updates, this last round of usability testing showed that the majority of usability issues on Places have been satisfactorily resolved.

JMinor added a comment.Jun 1 2017, 9:59 PM

Really great job @cmadeo , both on managing the process and reporting and acting on the findings.

It took a few go rounds, but the improvements in usability and user understanding were well worth it. So glad this kind iterative user feedback is now a viable part of our process.

JMinor closed this task as Resolved.Jun 1 2017, 9:59 PM
JMinor claimed this task.
JMinor awarded a token.

Thanks @JMinor! So thankful to be able to do this sort of testing.