Related links: Protocol | Analysis on Notion

Contents

1. TL;DR
2. Summary
3. Observations
4. Participants

1. TL;DR

• We’re going to move forward with Variant B, as expected usage of the feature is low, and to have one more shortcut in the toolbar.
• Most participants mentioned that they would not use the feature often; that’s why we keep changes to the existing version minimal (see T306709 for action items). We keep the way shortcuts are ordered (see the production version of the app).
• Similar to the debates we had in the team, there is no clear winner out of the two variants. The preference depends on what people are used to in the services/apps they use.
• We evaluate that the advantage of having space for another item in the toolbar in variant B outweighs the slightly better discoverability and the significant engineering effort of A.
• Answering the research questions from the protocol
  1. Do the updated tooltip designs with the ‘GOT IT’ option help participants find the new ‘Customize toolbar’ feature?
     • Yes, they do – we will keep the ‘GOT IT’ option.
  2. Do the changes in information architecture for both versions A and B help users find the ‘Customize toolbar’?
     • Yes, moving the menu item out of ‘Theme’ improved discoverability. We’re balancing this with the expected feature usage (see 6) and think the tooltip will serve its purpose.
  3. Do the additional default tooltips on the details help users understand the interface and what to do now?
     • Yes, almost all participants could drag and drop the items. However, we need to make sure they are correctly localized.
  4. Which variants perform better regarding users understanding how items in the toolbar connect to items in the overflow menu?
     • There’s a tendency in the test (6/10 participants) that variant A does better at connecting the toolbar and overflow menu. The finding will be balanced with the expected usage of the feature (see 6) and people already used to the existing way to change their toolbar icons.
  5. Is the updated language (shortcuts instead of ‘actions and destinations’) more understandable to users?
     • Yes, we will move forward with ‘shortcuts’ language.
  6. Gauge how participants expect the customize toolbar feature to work and how often they would want to use it.
     • Most participants would not use the feature often.

2. Summary and recommendations

• 🅰️ 6 out 10 participants prefer variant A, for the following reasons:
  • Overflow menu at the top reminds him of an internet page, would not go to look for it at the top if it wasn’t indicated, the bottom variant is less confusing, would be ok with B as well.
  • “It’s awesome”, ****it was easier to understand and find.
  • Likes this version better as it consolidate options in one place at the bottom.
  • The options are closer together, it works very well, people would benefit from the feature, maybe grouping could be helpful?.
  • It’s easy to understand how to find the customize option, is confused by variant B.
  • Expects the customization of the toolbar where the toolbar is, realizes that you loose 1 item in the toolbar in this version, but “you really don’t need more than 4 in the toolbar”.
• 🅱️ 4 out 10 participants prefer variant B, for the following reasons:
  • Literally suggested variant B during his test, prefers 1 more item in the toolbar, gives more flexibility.
  • One more link the bottom toolbar, a lot of people have a habit to click at the top menu to look for things.
  • It’s similar to Google Chrome.
• Most participants would not use the feature often. Here are reasons:
  • Would probably customize his toolbar just once or twice, and wouldn’t regularly use it again even though it’s a really good feature.
  • Would use the feature 2-3 times and thinks it would improve the experience.
  • Would use it once.
  • Thinks that not everyone would use the feature since a lot of people would use the default settings (20% of people would use it).
  • Would not change her links very often, maybe change it 1 or 2 time thing. (every few months), will likely need time to find out which are her favorites.
  • Would not use the feature that often, e.g. once a month or more, is confused by variant B.
  • Wouldn’t use it a lot, just uses it to switch the theme, only uses the ‘Language’ and ‘Find in page’ feature.
  • Wouldn’t mess with it too much but would use it until she figured out her preferences.
• Fix these translation issues:
  • Default tooltips in the ‘Customize toolbar’ detail screen are not translated in most languages.
  • Translate ‘Categories’ and ‘More’ into Arabic.
  • Hindi:
    • ‘Customize your toolbar’, ‘Edit history’ and ‘Categories’ are not translated.
    • Optimize ‘GOT IT’ translation in Hindi, participants had difficulties understand.
  • Japanese:
    • One participant mentions that the Japanese translation for ‘GOT IT’ means ‘ROGER THAT’. There might be a better translation (JA Male A #2)
    • Edit history is not translated.
• Arabic: ‘Customize toolbar’ entry point in the menu (overflow menu at the top or bottom sheet) are inverted in RTL languages. Needs to be fixed.
• People understand how to move things around in the ‘Customize toolbar’ detail screen, the default helper tooltips are working.
• Make sure app bar stays visible when the tooltip is shown, otherwise it points to nothing.
• Consider permanently showing the blue tooltip until the user taps ‘GOT IT’. Depends on how strongly we want to promote the feature.
• Move ‘Customize toolbar’ tooltip closer to the place where you can tap, it’s currently too far away from the 'More' item.
• Separator line above ‘Customize toolbar’ seems to be invisible in the black theme (ID Male A), needs to be verified and color adjusted.
• ‘Save’ and ‘Watch’ option are unclear/hard to distinguish for users (see HI Female B test)

3. Observations

AR Male A

• 😡 Tooltip points to an odd place at the bottom (in between Theme and More), needs to be optimized
• 😍 Sees and understands the ‘Customize toolbar’ tooltip
• 😡 Categories (in the article) and More (details screen) label are not translated to Arabic
• 😡 Doesn’t understand the the task to find customize the toolbar and clicks the article edit pencil first (10)
• 😍 Understands and uses the customize toolbar details screen
• 😍 Has no issues findings items moved out of the toolbar
• 😡 Would like to see more than 4 items in the toolbar, and suggests to move ‘More’ to the top right corner
• 😍 Summary: Mentions that it’s a very good feature
• 🅱️ Prefers variant B, literally suggested variant B during his test, prefers 1 more item in the toolbar, gives more flexibility. Would use the feature 2-3 times and thinks it would improve the experience

AR Female B

• 😡 App bar at the top disappears on scroll but tooltip is still shown
• 🤔 Has seen the tooltip but did not read it.
• 😡 Does not understand the ‘Customize toolbar’ task (10), has difficulties understanding the ‘links’ terminology
• 😡 The ‘Customize toolbar’ item in the overflow has not been optimized for RTL languages
• 😍 Understands and uses the customize toolbar details screen
• 😡 Confuses the TOC with Theme
• 😍 Summary: Mentions that it’s a really good feature for people that use Wikipedia on a daily basis, mentions that the ‘Black’ theme is too dark
• 🅱️ Prefers variant B, where the menu items were at the top, would use it once

FR Male A

• 😍 Does not understand the task that uses the word ‘tooltip’ but has actually red the tooltip and understood it
• 😍 Has no issues finding ‘Customize toolbar’
• 🤔 Calls the links in the toolbar ‘Quick buttons’
• 😍 Has no issues dragging and dropping items on the details screen. The default tooltips seem to help, moves the ‘Share’ button to his favorites
• 😍 Has no issues findings items moved out of the toolbar
• 😍 “Knowing that you care about your customers is very much appreciated!”
• 😍 Summary: Thinks it’s a good feature on Mobile, everything worked well and did not have any issues with the toolbar, would not change everything
• 🅰️ Prefers variant A, says that the overflow menu at the top reminds him of an internet page, would not go to look for it at the top if it wasn’t indicated, the bottom variant is less confusing, would be ok with B as well, would probably customize his toolbar just once or twice, and wouldn’t regularly use it again even though it’s a really good feature

FR Female B

• 😡 Taps the TOC and looks for ‘Customize toolbar’ and does not find it at the top at first
• 😍 Finds the ‘Customize toolbar’ screen after being prompted
• 😡 Does not understand the she needs to drag and drop items in order to reorder them, she clicks the items
• 😡 Does not understand where to access the items she moved out of the toolbar intuitively
• 😡 Summary: did not know how to drag & drop, would prefer clicking rather than dragging, the default tooltips did not help her
• 🅰️ Prefers variant A, “it’s awesome”, ****it was easier to understand and find

HI Male A

• 😍 Sees and understands the ‘Customize toolbar’ tooltip
• 😡 Does not understand the ‘GOT IT’ translation, can the string be optimized?
• 😡 Did not see that the tooltip points to the ‘More’ option in the toolbar. Maybe the tooltip’s too far away from the bottom toolbar.
• 😡 Does not know how to access ‘Customize toolbar’ until prompted
• 😍 Understands and uses the customize toolbar details screen, has no issues dragging and dropping
• 😍 Likes the ‘Customize toolbar’ screen
• 😍 Has no issues findings items moved out of the toolbar
• 😍 Summary: Very easy to understand. Likes the feature.
• 🅱️ Prefers variant B, since there’s one more link the bottom toolbar, a lot of people have a habit to click at the top menu to look for things, thinks that not everyone would use the feature since a lot of people would use the default settings (20% of people would use it), but thinks that the design is “very good” and he hasn’t seen this anywhere else yet, which is very good!

HI Female B

• 😍 Sees the ‘Customize toolbar’ tooltip.
• 😡 Does not understand what the tooltip means.
• 😍 Understands and uses the customize toolbar details screen, has no issues dragging and dropping
• 😡 Does not know what the difference between the star (’Watch’) and bookmark (’Save’) icon is
• 😍 Has no issues findings items moved out of the toolbar
• 😍 Loves the TOC swipe gesture a lot
• 😍 Likes the ‘Sepia’ theme color option a lot
• 😍 Summary: Likes the feature in general, “I haven’t missed this feature my entire life”, but does not like that the three dots in the app bar are disappearing when scrolling, very easy to reach and navigate, likes the division between the top and bottom toolbar
• 🅰️ Prefers variant A, likes this version better as it consolidate options in one place at the bottom, would not change her links very often, maybe change it 1 or 2 time thing (every few months), will likely need time to find out which are her favorites

ID Male A

• 😡 Does not see the tooltip (it’s visible through the entire test duration).
• 😡 Does not find the ‘Customize toolbar’ option
• 😍 Found the ‘Customize toolbar’ option after exploring some areas in the article.
• 😍 Understands and uses the customize toolbar details screen, has no issues dragging and dropping
• 😍 Has no issues findings items moved out of the toolbar
• 😍 Summary: Simple but effective feature, would not change much since it’s a comfortable experience
• 🅰️ Prefers variant A, since the options are closer together, it works very well, people would benefit from the feature, maybe grouping could be helpful?

ID Female B

• 😡 Does not see the tooltip, she tapped it away intuitvely
• 😡 Does not find the ‘Customize toolbar’ option
• 😍 Understands and uses the customize toolbar details screen, has no issues dragging and dropping
• 😍 Has no issues findings items moved out of the toolbar
• 😍 Summary: Everything is really easy, users would not know about the feature, it would be good if the customize toolbar feature would be visible in the toolbar itself.
• 🅰️ Prefers variant A, it’s really easy to understand how to find the customize option, would not use the feature that often, e.g. once a month or more, is confused by variant B

JA Male A

• 😍 Mentions that the tooltip shows new features in the app and understands what it means
• 😍 Finds the ‘Customize toolbar’ feature right away
• 😡 Edit history is not translated in Japanese
• 🤔 Does not see the point of actually customizing the toolbar, he would likely keep it as is
• 😍 Understands and uses the customize toolbar details screen, has no issues dragging and dropping
• 😍 Has no issues findings items moved out of the toolbar
• 😍 Likes ‘Reading focus mode’
• 😡 Title of the article stays in sans-serif, even though the serif option is chosen (maybe expected behavior)
• 🤔 Uses Wikiwand extension on Desktop
• 🤔 Summary: “I guess it’s nice but personally don’t see the point of it”, would not use the feature
• 🅱️ Prefers variant B, as it’s similar to Google Chrome, wouldn’t use it a lot, just uses it to switch the theme, only uses the ‘Language’ and ‘Find in page’ feature

JA, Female B

• 😡 Has seen the blue tooltip but tapped it away deliberately. Hasn’t read it.Assumed it was a “silly popup”
• 😡 Does not understand the terminology ‘bottom toolbar’ and ‘links’
• 😡 Does not find the ‘Customize toolbar’ option
• 😍 Understands and uses the customize toolbar details screen, has no issues dragging and dropping
• 😍 Has no issues findings items moved out of the toolbar
• 😍 Summary: It’s nice to customize the links, the limitation she see is that you need to have 5 links in the toolbar, is positively surprised that this is possible
• 🅰️ Prefers variant A, expects the customization of the toolbar where the toolbar is, realizes that you loose 1 item in the toolbar in this version, but “you really don’t need more in the toolbar”, wouldn’t mess with it too much but would use it until she figured out her preferences

4. Participants

• [✓] AR Male A
• [✓] AR Female B
• [✓] FR Male A
• [✓] FR Female B
• [✓] HI Male A
• [✓] HI Female B
• [✓] ID Male A
• [✓] ID Female B
• [✓] JA Male A
• [✓] JA, Female B [2]