Page MenuHomePhabricator

Conduct user testing to verify that new users will understand how to turn off Reading Mode
Closed, ResolvedPublic

Description

Background

The Android team is releasing a feature called reading mode that hides editing features. See details in T254771.
As we role this out this feature, it is important that users will know how to turn off reading mode should they desire to do so.


The Task

Run a test on usertesting.com in our target languages accounting for gender, geographic and age diversity, that tests if users are able to successfully turn reading mode on and off without much guidance

Note from Pau:

  • We will test if people know how to disable it again after enabling (and distracting them with another task between the enabling/disabling).

Evaluating the potential issue first makes sense. I guess the specifics for testing will be defined later, but one particular scenario that I think would be relevant to test would be this one: ask participants to correct a typo that's visible on a page after they have enabled the reading mode on a previous task before.

That would help to check if participants are able to connect the dots: looking for the edit pencil -> it is not there because I'm on reading mode -> find where to disable reading mode. This is a bit broader than just asking to disable the reading mode, making it about access to the features not available because of this mode.

Note from Jazmin:

  • Find out how accessible focus mode an customizable toolbar is via 'Theme' option

Analysis

👉 Check out the full version of the analysis on Notion

Recommendations
  1. Onboarding / language
    • Keep onboarding tooltips visible when scrolling. Only proactively tapping “GOT IT” should dismiss the tooltip.
  2. Reading focus mode
    • Make sure the Reading focus mode setting completely hides edit pencils, without having to refresh the screen (some participants experienced this)
    • ADD ARTICLE DESCRIPTION, ADD IMAGE CAPTION, ADD IMAGE TAGS CTA’s should be hidden when reading focus mode is enabled
  3. Customize toolbar
    • Make sure the entire list item is draggable not only the icon on the right. Alternative concepts if this is not feasible:
      • Show a tooltip when users try to drag and drop the label instead of the icon
      • If tooltip does not improve the situation → consider moving towards a pattern that does not rely on drag on dropping, e.g. numbered checkboxes
    • Terminology: Users don’t get the current wording ‘Actions and Destinations’, change the copy of the detail screen to: Customize the bottom toolbar with 5 quick links you find most helpful.
    • Discovery and usability (T302397)
      • Add an additional, sixth ‘More’ menu item to the bottom toolbar, since the feature is hard to discover within 'Theme' at the moment.
      • Move items from the overflow menu in the app bar (top right) to a bottom sheet that is triggered by the newly added sixth 'More' item in the toolbar.

Observations

What works?

  1. Onboarding / language
    • Tooltip is displayed for all participants
    • One candidate (FR) appreciated the full screen onboarding when editing article descriptions for the first time. This should be explored further in onboarding and guidance phase (T301397)
  2. Reading focus mode
    • Reading focus mode disabled by default as participants learned how it works
    • 4/5 understood what the reading focus mode setting does, understood differences when the mode is enabled
    • 5/5 had no issues going back to editing after turning on reading focus mode
    • 4/5 like the ‘Reading focus mode’ feature
  3. Customize toolbar
    • Once arrived in the customization screen, 5/5 understood what the screen does.
    • 5/5 figured out how to drag&drop items (but 3/5 were dragging the labels first)

What doesn’t work?

  1. General / Onboarding / Language
    • Tooltip wasn’t noticed by 4/5, participants scrolled and the tooltip disappeared
  2. Reading focus mode
    • Likely related to the tooltip that disappeared → 3/5 participants had issues discovering the ‘Reading focus mode’ setting
  3. Customize toolbar
    • Language: ‘actions and destinations’ in the task was confusing to almost all participants
      • All participants had issues finding ‘Customize toolbar’
      • Interaction: 3/5 participants wanted to drag&drop the label on the customization screen in order to move items.
      • 3/5 had issues finding items in the overflow menu after moving them out of the toolbar (they had troubles making a connection between the toolbar and overflow menu)

Event Timeline