Page MenuHomePhabricator

Usability test analysis for Add images MVP
Closed, ResolvedPublic

Description

Usability test analysis and recommendations

This is the analysis of the Android Add image AI trainer MVP: Research. This is an independent analysis, performed by the Android team and serves as a basis for discussion with the Growth team. The Growth team performed an additional analysis of the usability tests on usertesting.com.


  • Rank usefulness of information (rating scale)
  • Analyze tasks / identify patterns
  • Formulate / define action items
  • Process follow-up questions (RS)
  • Process follow-up questions (JT, RS, RH, MM)
  • Review recommendations (JT)
  • Prioritize recommendations and create Phab tasks (JT, RS)
  • Transfer findings to Mediawiki page (JT)

TOC

  1. TL;DR — Recommendations
  2. Analysis of tasks
  3. Analysis of rating scale
  4. Analysis of follow-up questions
  5. Appendix

1. TL;DR — Recommendations

  1. Optimize designs for smaller screens, as some participants had display issues on narrow viewports or with an increased text size setting on their device (accessibility feature). These are the suggestions based on the observations in the test:
    • The bottom sheet for image suggestions needs to be draggable in order to reveal the article contents below it. Also, participants tried to interact with the handle bar at the top of the bottom sheet. #needsDevelopment #mediumFeasibility
    • Also worth considering a max height of the bottom sheet in order to not cover the article completely. #needsDesign #lowFeasibility
    • Optimize tooltip positioning and handling on smaller screens, as they are cut off on smaller screens. #needsDevelopment #mediumFeasibility
  2. Optimize design for different themes. The tests revealed the following issues:
    • Create more suitable 'Train image algorithm' onboarding illustrations for all different themes. #needsDesign #highFeasibility
    • The checkbox items in the the 'No' and 'Not sure' dialogs have issues in dark/black theme and need to be optimized. #needsDesign #highFeasibility
    • The element of positive reinforcement/counter has displays in the dark/black theme and needs to be optimized. #needsDesign #highFeasibility
  3. Provide an easy way to access the entire article from the feed, e.g. by incorporating a 'Read more' link, tappable article title or showing the entire article right from the beginning. #needsDesign #highFeasibility
  4. Show success screen (see designs on Zeplin) when users complete daily goal (8/8 image suggestions) #needsDevelopment #highFeasibility
  5. Optimize the 'No' and 'Not sure' dialog copy to reflect that multiple options can be selected. Some participants weren’t aware that multiple reasons can be selected. #needsCopywriting #highFeasibility
  6. Optimize copy of the 'opt-in' onboarding screen, as there’s an unnecessary word at the moment ('We would you like (...)'). #needsCopywriting #highFeasibility
  7. Suppress “Sync reading list” dialog within Suggested edits as it’s distracting from the task at hand. #needsDevelopment #highFeasibility
  8. Show full image in the feed instead of a cropped version, as some participants had issues identifying the subject of the image because of a suboptimal crop (e.g. camera or rodent) #needsDesign #needsDevelopment #mediumFeasibility
  9. Incorporate pinch to zoom functionality, as participants tried to zoom the image directly from the image suggestions feed. #needsDevelopment #lowFeasibility
  10. Incorporate gesture to swipe back and forth between image suggestions in the feed, as participants were intuitively applying the gestures. Per previous conversation with @Dbrant, there are questions that need to be answered from a technical perspective. #needsDevelopment #mediumFeasibility
    • DB: Suppose the user swipes forward accidentally, and then immediately swipes back. Should that count as a 'Not sure' event?
      • RS: It should not count in order to not motivate people to do it.
    • DB: How do we know if the swipe was accidental or deliberate?
      • RS: Would it be possible to set a delay before counting it? -//JT: Explore the option of undo but you can only go back once. I'd prefer people not swipe forward.
  11. Optimizations for 'Suggestion reason' meta information
    • Optimize copy 'Suggestion reason' meta information as the current copy ('Found in the following Wiki: trwiki') is not clear enough. #needsDevelopment #needsCopywriting #highFeasibility
    • Might be worth to explore making the 'Suggestion reason' more prominent as participants rated its usefulness the lowest (likely due to low discoverability) #needsDesign #mediumFeasibility
  12. Optimize the discoverability of 'info' button at the top right as 2/5 participants had issues finding it. #needsDesign #mediumFeasibility
    • JT: What about a question mark instead of an i?
      • RS: Changing the icon won’t not solve the discoverability issue, I will consider adding an additional label or different positioning of the FAQ.
  13. Optimize design of positive reinforcement element/counter on the Suggested edits home screen, as it was positioned too close the task’s title. #needsDesign #highFeasibility
  14. Remove full screen overlay when transitioning to next image suggestion. This allows users to orient better and keep context after submitting an answer. #needsDesign #highFeasibility
    • JT/DB: If we remove the full screen overlay, where will we put the positive reinforcement?
      • RS: Same position as it is currently, but with less / zero opacity for the overlay
  15. Write FAQ page #needsPM #highFeasibility
  16. Explore tooltip "Got it" button #needsDevelopment #mediumFeasibility

2. Analysis of tasks

🥰 = Good — Participant had no issues
😡 = Bad — Participant had issues
🤔 = Not sure if good or bad — Participant might had difficulties understanding the question, did not explicitly interact with it or ignored the task completely
ParticipantTasks
#6#9#10#11#13#15#16#17#18#19#20#21#22#23#25#26#27#28#30
Battybrit🤔🥰🥰🥰🥰🥰🥰🥰🥰🤔🥰🥰🥰🥰🥰🤔🥰🥰🥰
brad.s🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰
147qb🤔🥰🥰😡🥰🤔🤔🥰😡🥰🤔🥰🥰🥰🥰🥰😡🥰🥰
TestMaster888🤔🥰🥰🥰🤔🥰🤔🥰🥰🥰🤔🥰🥰🥰🥰🥰🥰🥰🥰
Cherry 928😡🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🥰🤔🥰🥰
Overall evaluation😡🥰🥰🥰🥰🥰🤔🥰🥰🤔🥰🥰🥰🥰🥰🥰🤔🥰🥰
Onboarding and understanding of Suggested edits
Task #6 Do participants understand the tooltip? 😡
  • 2/5 discovered the tooltip but had issues understanding it.
  • 2/5 did not see the tooltip since it disappeared too quickly.
  • 1/5 discovered and understood the tooltip completely.
Task #9 Can participants explain the difference between tasks? 🥰
  • 5/5 were able to explain their understanding of the tasks in a sufficient way.
Task #10 Do participants understand what the 'Train image algorithm' task is all about? 🥰
  • 5/5 were able to describe the task in their own words well.
Task #11 What do participants associate with the robot icon? 🥰
  • 4/5 associated the robot icon with an algorithm, artificial intelligence (AI) or computer program
  • 1/5 didn’t know what it means
Train AI task - Onboarding and understanding
Task #13 Do participants understand the two onboarding screens? 🥰
  • 4/5 understand both onboarding screens.
  • 1/5 wasn’t reacting to the second onboarding screen (opt-in).
Task #15 How do participants interact with onboarding tooltips? 🥰
  • 3/5 understand the task due to the tooltips.
    • 1/5 mentioned that the tooltips are very helpful to understand the task.
  • 1/5 understands the task but did not pay attention to the tooltips.
  • 1/5 probably did not see or understand the tooltips.
Task #16 Is the tooltip copy clear enough? How’s the timing and positioning of the tooltips on various devices / screen sizes? 🤔
  • 3/5 read and understand the tooltip copy.
  • 2/5 did not interact with the tooltips.
  • 2/5 had tooltip display issues on a smaller phone.
  • 1/5 likes that the tooltip mentions the impact (help readers understand a topic)
Task #17 Do participants know what to do after all these onboarding measures? 🥰
  • 5/5 understand what to do now.
Train images task
Task #18 Do participants interact with the prototype naturally? 🥰
  • 4/5 are mostly comfortable interacting with the UI and make educated decisions.
  • 3/5 do not navigate to the file page without being prompted.
  • 2/5 navigate between the article and file page intuitively and without issues.
  • 1/5 is intimated to make decisions that affect Wikipedia articles, doesn’t know how to interact with the article (RS: possible due to small screen size) and doesn’t use file detail page intuitively.
Task #19 Do participants know how to navigate to the file detail page? 🥰
  • 5/5 successfully navigated to the file detail page after being prompted.
  • 1/5 tapped the 'info i' icon in the feed view first.
Task #20 How helpful is the meta information on the file detail page? 🥰
  • 3/5 consider the information on the file page as helpful.
  • 2/5 mention that the author is helpful.
  • 2/5 mention that the date is helpful.
  • 1/5 mentions that licensing info is helpful.
  • 1/5 mentions that the image description is helpful.
Task #21 Do participants know how to enlarge / zoom an image? 🥰
  • 5/5 tapped the image and used a pinch to zoom gesture to zoom the image.
  • 2/5 tried to zoom the image directly from the feed experience.
Task #22 Do participants know how to go back and forth between image suggestions? 🥰
  • 5/5 use swipe gestures to navigate back and forth between image suggestions.
  • 2/5 tapped the back button at the top left before using the swipe gesture.
  • 1/5 tapped the 'info i' button at the top right before using the swipe gesture.
Task #23 Do participants understand the 'Not sure' options? 🥰
  • 5/5 understand the 'Not sure' options.
  • 3/5 were selecting multiple reasons at once.
Task #25 Do participants understand the 'No' options? 🥰
  • 5/5 understand the 'Not sure' options.
Task #26 Do participants scroll or know how to reveal more of the article contents? 🥰
  • 4/5 were successful in scrolling the article to reveal more information
  • 2/5 wanted to use the pull indicator at the top of the image suggestion to reveal the article below before they scrolled the article
  • 2/5 tried to the tap the article title (1/5 scrolled afterwards)
  • 1/5 looked for a 'More' button to reveal more of the article’s content, then tapped the 'info i' button at the top right
Task #27 Do participants know how to access the FAQ? 🤔
  • 3/5 tap the 'info i' button at the top right to reveal the FAQ.
  • 1/5 explained that she would tap the back button and look for an FAQ there (RS: a possible way to success as there’s an FAQ section in the SE home screen)
  • 1/5 did not notice the 'info i' button at the top right
Task #28 How do participants interpret the element of positive reinforcement? 🥰
  • 5/5 understand what it is and identified the element as motivational, encouraging and/or daily goal
  • 1/5 wasn’t 100% sure about it but then identified it as a motivational element.
Task #30 Do participants notice the element of positive reinforcement that has been added to the card? 🥰
  • 5/5 participants identified the added progress indication in the card

3. Analysis of rating scale

1 = Not at all useful information
5 = Very useful information
ParticipantRating
First paragraph (#36)Description (#33)Filename (#32)Caption (#34)Suggestion reason (#35)
Battybrit55554
brad.s55453
147qb54343
TestMaster88854342
Cherry 92855522
Overall rating54.6442.8

4. Analysis of follow-up questions

1. How do you think the suggested images for articles are being found? And how would you rate the overall quality of the suggestions?
  • 5/5 mentioned that the images presented were relevant.
  • 4/5 associated the image suggestions with an algorithm or computer program.
  • 2/5 mentioned that the suggestions are associated with keywords.
  • 1/5 mentioned these are random suggestions.
2. Was there anything that you found frustrating or confusing, that you would like to change about the way this tool works?
  • 3/5 replied that it’s easy to use.
  • 1/5 that it’s tedious and cumbersome.
  • 1/5 suggested to show more than 1 image choice per article.
3. How easy or hard did you find this task of reviewing whether images suggested were a good match for articles?
  • 4/5 find it very easy to evaluate if it’s a good match for the article.
  • 1/5 think it’s hard and time consuming but well worth it.
4. Would you be interested in adding images to Wikipedia articles this way? Please explain why or why not.
  • 4/5 are interested in such a feature
  • 1/5 mentions he would not be interested
  • 1/5 mentions that she wants to know how accurate she is when reviewing images

5. Appendix

Detailed notes and evaluations per participant from usertesting.com.

Participant #1: Battybrit
Tasks

Task #1-5 (instructions)

Task #6 🤔
Tooltip disappeared before she was able to interact with it.

Task #7-8 (instructions)

Task #9 🥰
Understands the rough concept of it.

Task #10 🥰
Understands the concept of it.

Task #11 🥰
Associates the robot icon with AI.

Task #12 (instructions)

Task #13 🥰

  • Understands first onboarding screen.
  • Understands second onboarding screen.

Task #14 (instructions)

Task #15 🥰

Task #16 🥰

Task #17 🥰
Understands the tooltips and what to do now.

Task #18 🥰

  • Comfortable with using the interface
  • Does not use meta data intuitively
  • Notices the reach the goal message
  1. 🥰 Does not see a rodent on the image. Chooses low quality.
  2. 😡 Even though an aviator is shown, she chooses not relevant. Realizes that she got it wrong in the next one.
  3. 🤔 Not 100% sure if she’s correct, but chooses yes.
  4. 🥰 Is confident about the choice.
  5. 🥰 Makes informed decisions based on the image.
  6. 😡 Does not check meta data.
  7. 🥰 Not sure if a person is suited for a surname.
  8. 🥰 No, Not enough information
  9. 🥰 No, Not relevant
  10. 🥰 Not sure, Not enough information
  11. 🥰 Yes
  12. 🥰 No, Low quality
  13. 😡 No, Not enough information
  14. 🥰 Not sure, Not enough information
  15. 🥰 No, Not enough information
  16. 🥰 No, Not relevant
  17. 🥰 Not sure, Not enough information (different language)
  18. 🥰 Yes

Task #19 🤔

  • Taps the 'info i' button first.
  • Then figures out that she needs to tap the image itself.

Task #20 🥰

  • Mentions the info about the author, licensing and the date is helpful,

Task #21 🥰
Wants to zoom in the image by pinching (but it’s not yet implemented)

Task #22 🥰
Easily swipes to go back to the previous and next suggestions.

Task #23 🥰
Explains all options well.

Task #24 😡 (instructions)
Didn’t know that she can select more than one reason.

Task #25 🥰
Explains all options well.

Task #26 🤔

  • Would click a button to expand the article (RS: Not sure which button she’s referring to as there is none).
  • Then further explains that she would tap the article’s title.

Task #27 🥰
Would tap the 'info i' button to reveal more information

Task #28 🥰

  • Wasn’t 100% sure about it
  • Identifies it as a motivational element

Task #29 (instructions)

Task #30 🥰
Had no issues identifying what’s changed and what the additional number and bar means.

Task #31 (instructions)

Task #32-36 (see rating scale)

Task #37 (open format)
Answer/rating from other people that are using the feature to validate/correct her own choice.

Final thoughts
None

Follow-up questions
  1. I think they are being found by tags on the internet? maybe keywords? I would rate overall quality as good, some obviously were off but that makes sense since many places have similar names or multiple meanings.
  2. I didn't find anything confusing but at times I was disappointed with myself because I didn't know if one fit into a yes or no. I like to be accurate and sometimes didn't trust my own judgement.
  3. I found some very easy and clearly related or not related but some were tricky because I didn't know the subject well.
  4. I would be interested but as mentioned verbally, I'd like to know how accurate I am. I just would not want to be adding wrongly and would like to know my accuracy rate.

Participant #2: brad.s
Tasks

Task #1-5 (instructions)

Task #6 🥰
Understands the tooltip at the bottom.

Task #7-8 (instructions)

Task #9 🥰
Explains the 'Train image algorithm' task well and knows where images are positioned (on Desktop Wikipedia). Also knows what the others mean.

Task #10 🥰
Mostly understands the task.

Task #11 🥰
Associates robots with algorithms, even though he thinks that it doesn’t mean anything.

Task #12 (instructions)

Task #13 🥰

  • Understands first onboarding dialog.
  • Opts in to displaying his name on the algorithm training page.

Task #14 (instructions)

Task #15 🥰
Understands the task perfectly due to the tooltip. Says the tooltips are very helpful to understand the task.

Task #16 🥰
Likes that the tooltip mentions the impact that his choice has (help readers understand the topic)

Task #17 🥰
Mentions tooltips were very helpful. Similar to Google Maps feature (e.g. “does this shop have toilet facilities?”)

Task #18 🥰

  • Navigates intuitively between article and file details screen
  • Doesn’t consider 'Suggestion reason'
  • Selects multiple items in 'No' reason
  • Selected 'Not sure' only once
  1. 🥰 No, Not enough information — doesn’t check for the details of the image though
  2. 🥰 Yes, because of Filename
  3. 🥰 No, Not enough information — checks out the detail page of the image
  4. 😡 Yes (RS: I don’t think it’s suitable)
  5. 🤔 Yes
  6. 🥰 No, Not enough information
  7. 🥰 No, Not relevant
  8. 🥰 Yes
  9. 🥰 No, Not enough information, Don’t know the subject (RS: Article excerpt’s too short)
  10. 🥰 No, Not relevant, Don’t know this subject
  11. 🥰 Yes
  12. 🥰 No, Not relevant, Not enough information, Don’t know this subject
  13. 🥰 Yes
  14. 🥰 No, Other
  15. 🥰 No, Not relevant
  16. 🥰 No, Not relevant, Other
  17. 🥰 Not sure, Not enough information
  18. 🥰 Yes

Task #19 🥰
Intuitively taps on image card.

Task #20 🥰

Task #21 🥰
Zooms in the image on the detail page not in the main feed (RS: There are multiple paths to success).

Task #22 🥰
Easily swipes left and right between image suggestions.

Task #23 🥰
Explains all 'Not sure' options well.

Task #24 (instructions)

Task #25 🥰
Explains all 'No' options well.

Task #26 🥰

  • Knows how to scroll
  • Wants to use the pull indicator

Task #27 🥰
Would press the 'info i' icon to reveal more information

Task #28 🥰
Identified what the counter is at the bottom (encouragement).

Task #29 (instructions)

Task #30 🥰
Had no issues identifying wha the counter is.

Task #31 (instructions)

Task #32-36 (see rating scale)

Task #37 (open format)
Nothing to specific (audio broke up in the beginning)

Final thoughts

Follow-up questions
  1. using AI. the suggestions were very good and the majority were accurate.
  2. N/A
  3. very easy and user intuitive
  4. yes, as it is so easy and simple to do

Participant #3: 147qb
  • Uses quite a small phone
  • Uses accessibility setting that increases the font size in the app
Tasks

Task #1-5 (instructions)

Task #6 🤔
Reads it out loud and funny, not sure if he understood the tooltip though.

Task #7-8 (instructions)

Task #9 🥰
More or less describes the task’s correctly.

Task #10 🥰
Understands the task, doesn’t mention AI though.

Task #11 😡
Doesn’t know what it means.

Task #12 (instructions)

Task #13 🥰

  • Understands the first onboarding screen.
  • Identifies a copy error “We would you like (...)”

Task #14 (instructions)

Task #15 🤔
Not sure if he understands completely.

Task #16 🤔
Sees the tooltip.

Task #17 🥰
Taps 'Yes' accidentally but it seems that he understood the task.

Task #18 😡

  • Assumes that he needs to know about the article’s topic to make an educated decision if the image fits it or not.
  • Is intimated to make decisions for articles in their doctoral thesis's.
  • Does not interact with the article
  • Does not interact with the file detail page
  1. 😡 Not sure, Don’t know the subject
  2. 😡 Not sure, Not enough information
  3. 😡 Not sure, Not enough information
  4. 😡 Yes
  5. 😡 Yes
  6. 😡 Yes
  7. 😡 No, not enough information
  8. 😡 Not sure, Don’t know the subject
  9. 😡 No, Not enough information
  10. 😡 No, Don’t know the subject
  11. 😡 No, Don’t know the subject
  12. 😡 No, Don’t know the subject
  13. 😡 No, Don’t know the subject
  14. 😡 No, Low quality, Don’t know the subject
  15. 😡 No, Don’t know the subject
  16. 😡 No, Don’t know the subject
  17. 😡 No, Don’t know the subject
  18. 😡 Not sure, Not enough information
  19. 😡 No, Don’t know the subject
  20. 😡 No, Don’t know the subject

Task #19 🥰

  • Taps the image for more information
  • Mentions that he would use Google search to find more about the image.

Task #20 🤔

Task #21 🥰
Zooms the image on the detail page.

Task #22 🥰

  • Was looking for back functionality first
  • Swipes back and forth to navigate between suggestions.

Task #23 🥰

  • Understands the options presented
  • Understood that he can select multiple

Task #24 (instructions)

Task #25 🥰
Understand the options

Task #26 🥰
Realizes that he could scroll the article here

Task #27 😡
Didn’t see the 'info i' at the top right.

Task #28 🥰
More or less understands the concept

Task #29 (instructions)

Task #30 🥰
Notices the added counter

Task #31 (instructions)

Task #32-36 (see rating scale)

Task #37 (open format)

Final thoughts

Follow-up questions
  1. it's a random thing and I would say that the overall quality is good.it's a random thing and I would say that the overall quality is good.
  2. it's pretty tedious, and it's pretty cumbersome. I guess it has to be because Wikipedia is something that is very serious, and I think it should be taken seriously.
  3. pretty hard but time consuming and I would have to say it was well worth it
  4. probably not is that I have a weak bladder and I have to use the men's room much much much too often to be a good at any of this. I am sorry take care

Participant #4: TestMaster888
  • Did not notice the 'Suggestion reason'
Tasks

Task #1-5 (instructions)

Task #6 🤔
Doesn’t see the blue tooltip.

Task #7-8 (instructions)

Task #9 🥰
Describes the tasks properly.

Task #10 🥰
Describes the task properly.

Task #11 🥰
Associates the robot with a computer program / algorithm.

Task #12 (instructions)

Task #13 🤔

  • Seems like he understands the first onboarding dialog
  • Doesn’t react to the second onboarding dialog

Task #14 (instructions)

Task #15 🥰

  • Describes what he sees (article and image)
  • Hasn’t really paid attention to the tooltips

Task #16 🤔
Hasn’t really read the tooltips.

Task #17 🥰
Knows what he needs to do now.

Task #18 🥰

  • Makes educated choices
  • Reads meta information of the image
  • Zooms the image intuitively
  • Navigates back and forth on detail page and article
  • Makes educated decisions
  1. 🥰 No, Not relevant
  2. 🥰 Yes
  3. 🥰 Not sure, Not enough information
  4. 🥰 No, Not relevant
  5. 🤔 Yes
  6. 🤔 Yes
  7. 🤔 Yes
  8. 🥰 Yes
  9. 🤔 Yes
  10. 🥰 No, Not relevant
  11. 🥰 Yes
  12. 🥰 Not sure, Not enough information
  13. 🥰 Yes
  14. 🥰 No, Not relevant, Low quality
  15. 🥰 Not sure, Not enough information
  16. 🥰 No, Low quality
  17. 🤔 Yes
  18. 🥰 Yes
  19. 🥰 Not sure, Not enough information
  20. 🥰 Yes

Task #19 🥰

  • Uses file detail page intuitively
  • Mentions that he’d Google the image

Task #20 🤔

Task #21 🥰
Used pinch to zoom intuitively

Task #22 🥰

  • Taps the button at the top right first
  • Then swipes between suggestions

Task #23 🥰
Explains the options well

Task #24 (instructions)

Task #25 🥰
Explains the options well

Task #26 🥰
Scrolls down with no issues

Task #27 🥰
Taps on the 'info i' icon at the top right

Task #28 🥰
Identifies it as an element of positive reinforcement

Task #29 (instructions)

Task #30 🥰
Sees that the counter has been added

Task #31 (instructions)

Task #32-36 (see rating scale)

Task #37 (open format)

Final thoughts

Follow-up questions
  1. The algorithm is programmed to look at keywords in the article and do an image search based on the keywords and try to return an image that best matches its search parameters. 65% the images I was presented were relevant.
  2. Nothing was confusing. though it might work better if the algorithm presented the human editor with more than 1 image choice.
  3. very easy.
  4. Yes it would be much better than scouring the web for images manually.

Participant #5: Cherry928
Tasks

Task #1-5 (instructions)

Task #6 😡

  • Describes the Explore Feed card
  • Reads out the tooltip loud but does not understand it

Task #7-8 (instructions)

Task #9 🥰
Understands the tasks

Task #10 🥰
Understands the task

Task #11 🥰
Associates the robot icon with an algorithm.

Task #12 (instructions)

Task #13 🥰
Understands both onboarding screens

Task #14 (instructions)

Task #15 🥰
Sees onboarding tooltip and knows what to do.

Task #16 🥰
Reads tooltips and understands what to do.

Task #17 🥰
Knows what do now.

Task #18 🥰

  • Does not interact with file detail page at all, probably due to time pressure (mentions that these tests should only be 15mins long)
  • Analyzes the information she sees thoroughly
  • Scrolls the article intuitively
  • “Would you add this image to the article?” is cut off → Display issues on smaller screens
  • Selects multiple answers right away
  1. 🥰 No, Not relevant
  2. 🥰 Yes
  3. 🥰 Not sure, Not enough information
  4. 🤔 Yes
  5. 🥰 Not sure, Not enough information
  6. 🥰 Not sure, Not enough information
  7. 🥰 Yes
  8. 🤔 Not sure, Other
  9. 🤔 Not sure, Not enough information
  10. 🥰 Not sure, Not enough information
  11. 🤔 Yes
  12. 🥰 Not sure, Other
  13. 🥰 Yes
  14. 🥰 No, Low quality
  15. 🥰 No, Not relevant
  16. 🥰 Not sure, Not enough information
  17. 🤔 Yes
  18. 🥰 Yes
  19. 🥰 Not sure, Not enough information, Don’t know this subject

Task #19 🥰
Taps on the image card

Task #20 🥰

  • Date taken
  • Read full image description
  • Author (to contact them)

Task #21 🥰

  • Tries zooming the image from the feed
  • Taps on the image and zooms in on detail page

Task #22 🥰
Uses swipes gestures

Task #23 🥰
Understands the options

Task #24 (instructions)

Task #25 🥰
Understands the options

Task #26 🥰

  • Tries to tap the article title
  • Also tries to scroll the article
  • Taps the info bubble as the last step

Task #27 🤔
Would tap the back button and look for a help page (RS: A possible way to success)

Task #28 🥰
Likely understands it

Task #29 (instructions)

Task #30 🥰
Sees it and likely understands it

Task #31 (instructions)

Task #32-36 (see rating scale)

Task #37 (open format)

  • Easy access to article
  • More image information

Final thoughts

Follow-up questions
  1. By using key words found in the article title/summary or text. Most suggestions were definitely or likely related, so very good.
  2. Easy access to full articles. Description of image shown in full/more of.
  3. Fairly easy.
  4. Possibly if I had a lot of time on my hands, but probably not in my current situation as I have other things to spend my time on. But if someone is interested in contributing in a fun/easy way, I think this is a good way to do so.

Details

Due Date
Mar 24 2021, 7:00 AM

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
scblr renamed this task from Usability test findings and action items to Usability test analysis and action items.Mar 19 2021, 12:14 PM
scblr updated the task description. (Show Details)
LGoto set Due Date to Mar 24 2021, 7:00 AM.Mar 22 2021, 8:46 PM
scblr updated the task description. (Show Details)
scblr added a subscriber: Dbrant.
scblr updated the task description. (Show Details)
scblr renamed this task from Usability test analysis and action items to Usability test analysis and recommendations.Mar 23 2021, 4:06 PM
scblr updated the task description. (Show Details)
scblr updated the task description. (Show Details)

Hey @schoenbaechler, Dmitry and I went through this together and made updates

After our discussion with @MMiller_WMF and @RHo, we will incorporate agreed upon suggestions from the recommendations document and leverage the screenshots in their deck. Once that is incorporated, I will create subtasks and prioritize them on the Release Board.

scblr renamed this task from Usability test analysis and recommendations to Usability test analysis for Add images MVP.Mar 24 2021, 9:18 AM
Adding the items from Growth’s spreadsheet here in order to have one source of truth:
  1. Suggestion reason text: "Make this more prominent on the main screen.Make the labels more easily understood by showing the full Wikipedia language/project name and more lay-person friendly descriptions of the Wikidata and commons category reasons
    • Agreed but Dependency Exists. Desire to have it with text from robot. Separate out the Suggestion Reason text.
  2. Article preview: Show the entire article, or else provide affordance that there is more to the article to read (e.g., "Read more" link)
    • Done; Instrument this
  3. Image metadata under the image on the main screen: "Make it clear that reviewing the image metadata is a core part of the task. We can potentially do that by increasing the visual prominence and/or increase the affordance to promote always opening the metadata screen.
    • Lets Discuss
  4. Unsure reason: "Make it clear that reviewing the image metadata is a core part of the task. We can potentially do that by increasing the visual prominence and/or increase the affordance to promote always opening the metadata screen.
    • Sure
  5. More details screen: Provide affordance when descriptions etc have been truncated (eg., more info screen on the Caldwell image)
    • Lets Discuss and see examples
  6. Image cropping: Show the full image contained instead of a cropped image
    • Sure; Robin has an idea for this (Fit content)
  7. More details screen: Include the same metadata in the card - notably the suggestion reason (but also filename, image description and caption) on the more details screen as well.
    • Lets Discuss: Details are pulled from Commons, so this would be challenging and would require a hack
  8. Black and Dark mode display issues: Ensure all new screens display as expected when the user is in Dark or Black mode (Text and illustrations)
    • Sure
  9. Navigating back: Save previous answer state: Given users are able to go back, the selection made in the previous image or images should be retained
    • Lets Discuss the approach on what this actually look like
  10. Image information: Provide clear information that images come from Commons, or some more overt message about the image source and access to more metadata
    • Lets Discuss; Are we talking about the main screen or when people navigate to more info? Would it make sense to call this out in onboarding? Have the Wikimedia Commons logo on the more details page?
  11. Daily goal reached screen: Incorporate this element, missing in the prototype, into the MVP.
    • Sure
  12. All screens: Adjust layouts gracefully for when text overflows due to device size, default font size, or language line length differences
    • Sure
  13. More details screen: Reduce the font-size of the fields
    • Sure; Robin will have to provide new font size/definition
  14. Goal count: Messaging that lets the person know they can keep going past the daily goal count?
    • Lets Discuss: We are going to have the goal set to 10 mins and then check in at 2 weeks
  15. Edits screen: Separate the train image algorithm from the other suggested edits
    • Lets Discuss: I don't recall seeing this as a problem in the user testing videos
  16. Navigating back: Provide some visual affordance that users can go back and forward beyond the swipe gesture.
    • Lets Discuss: Maybe for going back but we need to talk about how this impacts analytics. We should talk about the tradeoff of an undo button that goes backwards one or being better about checking that users are sure about their choice
  17. Discovery: Add some more prominent and less ephemeral introduction to the new Suggested edit type than the blue tooltip on the Explore fee. Potential ideas are a blue dot or other visual indicator on the 'Edit' menu item itself, and/or a card on the explore feed.
    • Lets Discuss: We talked about having a Got it button on the tooltip to be more explicit
  18. Onboarding: Shorten the message on the full screen onboarding step 1, to concentrate on a single point that the task is to train AI in matching whether images are good for Wikipedia articles.
    • Maybe; We can include this as a low priority task.
  19. Unexpected unrelated Reading list sync dialogs appearing on Suggested edits and Train images screens: Suppress ""Turn on reading list sync" dialogs from appearing when the user is on the SE screens
    • Sure
  20. Onboarding: onboarding step 2 text has a typo - remove the extra "you" in "We would you like..."
    • Sure
  21. Image zooming: Provide affordance that the image can be zoomed
    • To be clear this means the detailed screen? If so then Yes

CC @JTannerWMF