Page MenuHomePhabricator

Design for Add images usability tests
Open, HighPublic

Description

👉 Design and conversations were mostly happening Add images for Android slide deck.


Image recommendations for Android

Table of contents

  1. Why are we doing this?
  2. User stories
  3. Concept
  4. Future
  5. Appendix

1. Why are we doing this?

👉 From the [Image Recommendations MVP document.

The Android, Structured Data, and Growth teams aim to offer image recommendations as a “structured task”. More about the motivations for pursuing this project can be found in the 4. IR_Planning & Spec document. In order to roll out Image Recommendations and have the output of the task show up on wiki, a MVP needs to be created to enhance the algorithm provided by the research team and answer questions about the behavior of our users to determine the potential success and needed improvements of this project.

2. User stories

2.1. Discovery

When I am using the Wikipedia Android app, am logged in,
and discover a tooltip about a new edit feature,
I want to be educated about the task,
so I can consider trying it out.

2.2. Education

When I want to try out the image recommendations feature,
I want to be educated about the task,
so my expectations are set correctly.

2.3. Adding images

When I use the image recommendations feature,
I want to see articles without an image,
I want to be presented with a suitable image,
so I can select images to add to multiple articles in a row.

2.4. Positive reinforcement

When I use the image recommendations feature,
I want feedback/encouragement that what I am doing is right/helping,
so that I am motivated to do more.

3. Concept for MVP (👉 Link to Zeplin ↗)

3.1. Discovery

  • A tooltip for logged in users is displayed to inform people that are using the app about the newly added “Add images” task.
  • (to be considered) Enhance current tooltip design with a “Got it” option to make dismissing the tooltip more obvious.

3.2. Education

  • A “Prototype” indication informs editors that the newly added task is different than the existing one
  • Further education happens via feature onboarding screens. Consider the copy and designs above as drafts. It’s also possible to add multiple slides to the onboarding to not overwhelm editors with information.

3.3. Adding images

3.4. Positive reinforcement (Link to research deck ↗)

We’re going to focus on positive reinforcements on the client side to reduce technical overhead within the scope of the MVP. Despite the limitations, elements of positive reinforcement play a vital part in the project, as users contribute in an invisible way – they’re training the image matching algorithm. This chapter lists design experiments for client side elements of positive reinforcement mechanism. To be discussed with the PM and Android Engineering lead. Disclaimer: The visuals below are drafts to illustrate the idea.

3.4.1. Animation and sound (Slide 70 ↗)

After adding a certain amount of images, (a) success animation(s), paired with a motivational messaging via a snackbar are displayed. The messaging could e.g. be used to explain how the user’s contributions are used for. Well-crafted copy is essential for this type of positive reinforcement.

3.4.2. Progress indication / levels (Slide 50 ↗)

Progress bars and levels help in giving users a sense of achievement for their “invisible” contributions. This design exploration proposes to introduce levels, specific to the image recommendation task. It indicates users how many steps are left to reach a level. Users could be rewarded with a success animation once they reached a level, e.g. with the mechanism outlined in 3.4.1.

3.4.3. Social connection (Slide 12 ↗)

Research on online communities indicates people are more likely to start participating if they see that their work will be seen, and in turn engender a sense of connection.

In this experiment, an indication of how many people have added contributions is added to the Suggested edits home screen. Assumably, this will need a server side query to determine how many unique users / or contributions have been made. Alternatively, this can be introduced as a global indicator in the profile stats section at the top.

3.4.4. Awards (Slide 14 ↗)

Cultivate reputations – within Wikipedia via showcasing their expertise and credibility on self-curated user profile pages.

This concept requires design work for awards that are related to the “Add images” task. It can be pursued independently or combined with the positive reinforcement mechanisms outlined in 3.4.1. and/or 3.4.2. To be considered: native user profile pages do not exist yet in the Wikipedia Android app.

4. Future (Beyond MVP)

4.1. Future user stories

4.1.1. Article experience

When I read an article on Wikipedia,
I want to be notified when there’s no image in this article,
I want to be presented with a suitable image for this article, 
so I can select an image to add to this article.

4.1.2. Explore feed card

When I’m browsing the Explore feed,
I want to see articles with no image,
I want to be presented with a suitable image,
so I can select an image to add to this article.

4.1.3. Connecting different types of tasks

When I successfully used the image recommendations feature,
I want to be presented with a way to add further information,
So I can add a suitable caption or tag to the image.

5. Appendix

5.1. Todos / questions

INITIAL TODOS

  • Study Growth’s prototype
  • Study Growth’s research plan
  • Study Growth’s usability test findings (RS, RH)
  • Lay out options for positive reinforcements on the client side (RS)
  • Refine user stories to be more from the user point of view, based on RH’s feedback (RS)
  • Add visual design directions based on user stories (RS)
    • Discovery
    • Education
    • Adding images
    • Add “I’m not sure" options to designs
      • Design flow for “More” to get to Commons file page

FEEDBACK BASED ON PRESENTATION ON FEB 3, 2021

  • Add conceptual descriptions to design directions (RS)
  • Create slide deck to communicate ideas better
  • Add more metadata to image suggestions (Filename, Description, Caption, Source) [MM, RH]
  • Improve integration of suggestion in the article, see comment [MW, JT]
  • Reduce “typing effect” in designs after first decision [RS]
  • Remove user profile icon from designs [RH]
  • Consider changing thumbs up / down to checkmark and X OR no icons at all [MM]
  • Disabled instead of progressive state of button [RS]
  • Visualize fixed “minimized” header on scroll [RS, RH]
→ RS: Bottom sheet approach feels more appropriate, as a fixed element at the top would collide with the scrollable article and conversation
  • Visualize a progress bar of some kind [JT]
  • Explore alternatives to confetti in success message [RS, RH]
  • “No / Not sure” related
    • Checkboxes instead of radio buttons to allow multiple answers, mandatory at least 1 though [MR]
    • Describe what happens when tapping “Not sure” (displayed every fifth time) [RS, JT]
    • Describe what happen when tapping “No” (displayed every time) [RS, JT]
  • Consider adding “Next” & “Previous” buttons [JT], write down pros and cons
→ RS: Overloads the interface, do we really want people to skip suggestions?
  • Onboarding
    • Visualize contextual onboarding with bot and/or tooltips [JT]
    • Split onboarding into three parts, study “Add a link” onboarding from Growth [RH]
  • Update slide deck: Add images for Android - Google Slides
  • Language
    • Rename “Submit” to “Continue” [JT]
    • Prototype, is it the right word? Alternatives:
      • Algorithm gym, training, ... [RH]
    • Optimize language for “is this good image for the article”. What if it said
      • Would a reader want to see this image in the article? [AS]
      • Would you want to see this image in the article? [AS]
      • Is this image helpful to understand the article? [JJ]
      • Does this image help explain the subject? [JJ]
      • Does this image help you understand the article? [JJ]
      • Does this image help the reader understand the article? [JJ]
  • Explore workflow for blind users
  • Formulate analytics questions, together with JT and SN [RS]

5.2. Related links

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 7 2020, 3:26 PM
LGoto triaged this task as Medium priority.Dec 7 2020, 5:08 PM
LGoto moved this task from Needs Triage to UX Debt Backlog on the Wikipedia-Android-App-Backlog board.
schoenbaechler moved this task from Backlog to Doing on the Android Design board.Jan 21 2021, 11:05 AM
schoenbaechler added a comment.EditedJan 21 2021, 11:19 AM
Notes from conversations
2020-12-07 | Dmitry Brant
  • Miriam is continuing to go through the updates.
  • Mixed messages from services team re: timelines when it would be ready (something ready or not ready in Q3).
  • Service might not be available in Q3.
  • Where do we insert the image in the article that get suggested?
    • DB: Suggests to insert image at the top
    • If we always insert it at the top, the community is fine with it
      • Feature parity with visual editor
    • Image in articles have caption
      • Is the user also supposed to add that caption
      • Adding caption is the next flow
schoenbaechler added a comment.EditedJan 21 2021, 11:20 AM
Notes from conversations
2020-12-07 | Dmitry Brant
  • DB: Miriam created a dump spreadsheet that we can use to create a prototype.
  • DB and RS agree on the direction → map and collect ideas for Android, then roughly sketch them out until the end of this week (or end of year since RS is working in the week of Dec 28)
schoenbaechler added a comment.EditedJan 21 2021, 11:20 AM
Notes from conversations
2020-12-16 | Pau Giner
  • PG encourages to think about how to connect different tasks to keep people engaged.
  • PG advises to map/draw different directions for IR on Android
    • Compile behavior and ideas of a new project
    • His process is to map items into motivation, discovery, etc., here’s an example Trello board for multilingual editors
    • RS thinks about an exercise to create a mindmap and then group / sort ideas around it.
  • RS mentions cross collaboration with Jess to talk about image insertion in visual editor
  • PG: Research in this are? Learn from people that are adding images (Rita)
schoenbaechler added a comment.EditedJan 21 2021, 11:21 AM
Notes from conversations
2021-01-13 | Jazmin Tanner
  • Explore interaction model of dating apps (swipe left and right?)
  • How do we deal with languages?
    • Do we support any other language than English Wiki?
    • Do we want to showcase images that have a caption in other (app) languages?
    • Options to choose language?
    • Caption language?
  • Onboarding to image recommendations?
  • Survey for image recommendations? How do we display the survey
schoenbaechler added a comment.EditedJan 21 2021, 11:22 AM
Notes from conversations
2021-01-19 | Rita Ho
  • The aim of the Suggested edits feed experience is to help improve the algorhithm
  • Is it then a demo?
    • “Algorhithm Gym”
    • Not disappointed and confused by not making an edit.
    • Separate version of adding images.
    • Positive reinforcements since users don’t edit Wikipedia (Trainer like crowdsource).
    • Contributions are not counted.
  • Data
    • People looked at image info a lot in the usablity tests.
    • Difficulty of adressing newcomers (Wikipedia vs Commons)
    • Find out if the feature is for more advanced editors on Android.
  • Usability test, two variants
    • Optional captions were part of the test.
    • After they did 20 imagre recommendations they had to add few (optional) caption in the end.
    • Captions were really bad.
      • Do we want to include these?
schoenbaechler added a comment.EditedJan 21 2021, 11:23 AM
Notes from conversations
2021-01-19 | Chat with Growth
  • RS: Can we find out which meta data is the most important?
  • RH: Convey algorithm gym task aspect
  • MM: Stats or credit on progress of the task. Positive reinforcements.
  • RH: Prioritize infos
  • MM: Convey that image is not there yet. RS will explore directions for this.
  • RH: Audience is bigger on Android. A lot of the metadata is written in English. Figure out how to deal with languages.
  • RH: Consider diary study from Carolyn on iOS analytics data (“Appsy”).
  • Additional links from Rita (Jan 20) per Slack message:
schoenbaechler renamed this task from Design explorations for IR to Design explorations for image recommendations.Jan 21 2021, 1:30 PM
schoenbaechler updated the task description. (Show Details)
schoenbaechler updated the task description. (Show Details)

@Dbrant @JTannerWMF

Per our discussion from Wednesday, I outlined a few possible experiments on positive reinforcements for the “Add images” task. The suggestions base on a broader research piece.

👉Check out chapter 3.4. in the task’s description and please review it / leave comments before our next sync meeting.

@RHo had a chat about elements of positive reinforcement yesterday. Adding notes from our conversation:

  • RH suggests a lightweight element (notifications / thanks / message), that is contained to the “Add images” task to avoid overhead
    • It’d probably make sense to build on notifications, since they proved to work
      • Put effort into crafting the message (RS, note to self)
      • Are the old ones still in place? (RS, needs to check)
      • Try out different notification messaging with A/B testing (RS)
  • RH has the feeling that we’re pinning to much on to image recommendations
    • The main purpose of the task is to train the tool
    • How would we test if a specific element of positive reinforcement is successful? A/B test with badges? Does it give an answer if the specific positive reinforcement tactics work or not?
  • About a notification when the new feature is available (new sell)
    • RH and RS are both on the same page that it might feel too intrusive.
  • RH: Is continuous usage of the tool important? As it’s a prototype to train the algorithm?

@RHo, feel free to unsubscribe from the task again after reading this. I know, too many notifications are unhealthy (ironically to the topic of our conversation yesterday 🙃)

RHo added a comment.Jan 28 2021, 12:19 PM

Thanks @schoenbaechler for capturing notes here. I added a few additional notes below but this all sounds right.

@RHo had a chat about elements of positive reinforcement yesterday. Adding notes from our conversation:

  • RH suggests a lightweight element (notifications / thanks / message), that is contained to the “Add images” task to avoid overhead

Potentially just a count of number of images reviewed within the "Add images" task if we want people to track how many they've done, but making sure it is separate from the main "Contributions" section.

  • RH has the feeling that we’re pinning too much on to image recommendations
    • The main purpose of the task is to train the tool
    • How would we test if a specific element of positive reinforcement is successful? A/B test with badges? Does it give an answer if the specific positive reinforcement tactics work or not?

Yes, these were really rhetorical questions as I think it would be diluting the focus if positive reinforcement tactics are also being tested at the same time as this initial image matching MVP.

  • About a notification when the new feature is available (new sell)
    • RH and RS are both on the same page that it might feel too intrusive.
  • RH: Is continuous usage of the tool important? As it’s a prototype to train the algorithm?

Along the same lines as above, my concern is that by providing specific incentives like awards/badges for working on something that is a temporary contribution type not resulting in an edit, it may be a confusing message to send, esp. for something that located on the "Edits" part of the app.


@RHo, feel free to unsubscribe from the task again after reading this. I know, too many notifications are unhealthy (ironically to the topic of our conversation yesterday 🙃)

Thanks @schoenbaechler - all good!

schoenbaechler renamed this task from Design explorations for image recommendations to Design for image recommendations.Jan 28 2021, 6:16 PM

Hey @schoenbaechler a thought I had for social connection is to show what percentile the person is in the for the day as it relates to task completion as opposed to how many people have contributed. Not sure if that is technically possible @Dbrant ?

Notes from the conversation with @JTannerWMF earlier today:
  • RS and JT are going to create a slide deck for the “Image Recommendations MVP Sync” meeting on Wed, Feb 3, 2021
  • JT mentions to go through the questions T273055 and add decisions / recommendations to it
  • JT is going to create a project page for image recommendation on Android and RS will add thoughts from a design perspective to it.
  • Copy suggestions for description in the main screen
    • Help us understand how well these images match with Wikipedia articles.
    • Help us improve the image matching algorithm.
    • Help to improve image matching suggestions for articles.
    • Help us train artificial intelligence to suggest matching images to articles.
  • RS mentions UX writing for machine-assisted editing deck, which is interesting to study for crafting the copy for the task. JT will have a look at it.
  • Onboarding
    • Both agree to make the onboarding dialog’s text more succinct.
    • JT suggests tutorial onboarding, RS recommends contextual onboarding with improved tooltips (that include “GOT IT” to dismiss them)
  • RS will create visual for JT’s “Tinder” swipe right (yes) and left (no) idea
  • JT and RS will define what happens when users taps, “Yes”, “No” and “Not sure” in detail in the deck
Notes from the recent conversation with @Dbrant:
  • DB: Consider to make the image larger (full width)
  • DB: Provides more info in regards to “Not sure”, “No”, ”Yes” options
    • If users press “Yes”, we pass the data to improve algorithm.
    • The “No” or “Not sure” dropdown should provide reasons for not being able to complete the task:
      • Image is plainly wrong
      • Subtly wrong (slight difference)
      • Caption in the wrong language (“I don’t understand the caption”)
      • Maybe one or two other choices (tbd)
    • DB mentions that the most simple pattern to present these options is a modal dialog dialog
      • RS shows options to make this more in context, e.g. by presenting message options to reply
    • Chat about positive reinforcement concepts in the task. DB’s feedback:
      • “X people contributed“ is not possible (we don’t have that number)
        • There are maybe possibilities for it but not for the first iteration (needs server side technology and can be done in a future quarter)
      • Levels
        • Need server side technology as well
      • Award
        • Adding an award to yourself is feels unnatural to Wikipedia
        • Barnstars are added by others
        • External account should award you for the edits
RHo added a comment.Tue, Feb 2, 7:53 AM
Notes from the recent conversation with @Dbrant:
  • DB: Consider to make the image larger (full width)
  • DB: Provides more info in regards to “Not sure”, “No”, ”Yes” options
    • If users press “Yes”, we pass the data to improve algorithm.
    • The “No” or “Not sure” dropdown should provide reasons for not being able to complete the task:
      • Image is plainly wrong
      • Subtly wrong (slight difference)
      • Caption in the wrong language (“I don’t understand the caption”)
      • Maybe one or two other choices (tbd)
    • DB mentions that the most simple pattern to present these options is a modal dialog dialog
      • RS shows options to make this more in context, e.g. by presenting message options to reply

Hi @schoenbaechler - some more relevant info on the "No reasons" selection.
On the "Add a link" structured task, we are showing a dialog with multiple options relevant to why a user would reject the suggestion as per DB's note.
You can see the task here: https://phabricator.wikimedia.org/T269647
Notes:

  • While the reasons for the image not matching would be different from links, I would recommend to always provide an "Other" option.
  • We are making the rejection reason optional, but I think you could make it mandatory so long as there is an "Other" option available
  • The user should be able to change their reason after selection (in case of wrong tap, etc) so the dialog should be able to be brought back after selection.

The root ticket T268704 has a links to a mobile and desktop prototype in case you'd like to play with the whole Add links flow.

  • Chat about positive reinforcement concepts in the task. DB’s feedback:
    • “X people contributed“ is not possible (we don’t have that number)
      • There are maybe possibilities for it but not for the first iteration (needs server side technology and can be done in a future quarter)
    • Levels
      • Need server side technology as well
    • Award
      • Adding an award to yourself is feels unnatural to Wikipedia
      • Barnstars are added by others
      • External account should award you for the edits

It does seems like the most apt and feasible positive reinforcement option is to show a count of image suggestions evaluated by the user. I do think these award and recognition types we discussed in the Positive Reinforcements research as being more appropriate to consider for more "consequential" actual article edits down the track.

schoenbaechler updated the task description. (Show Details)
JTannerWMF added a subscriber: Miriam.

From @Miriam :

The image is not depicting the subject of the article
The image is of very low quality
The image is relevant, but we would not use it on our Wiki
There is not enough metadata to understand the goodness of the image match
I am not expert enough on the article subject to judge the goodness of the image match
The image is just a placeholder for missing images
The image is not available
I did not understand the task

If you think 8 options are too many for this use case, we could skip/merge number (6) and (7), as they are just a special case for (1), and maybe merge (4) and (5).
Please let me know your thoughts, I am happy to iterate over those as much as needed.

JTannerWMF raised the priority of this task from Medium to High.Tue, Feb 9, 4:25 PM
JTannerWMF moved this task from To Do to In Progress on the Image-Recommendations board.
RHo added a comment.Tue, Feb 9, 5:45 PM

From @Miriam :

The image is not depicting the subject of the article
The image is of very low quality
The image is relevant, but we would not use it on our Wiki
There is not enough metadata to understand the goodness of the image match
I am not expert enough on the article subject to judge the goodness of the image match
The image is just a placeholder for missing images
The image is not available
I did not understand the task

If you think 8 options are too many for this use case, we could skip/merge number (6) and (7), as they are just a special case for (1), and maybe merge (4) and (5).
Please let me know your thoughts, I am happy to iterate over those as much as needed.

+1 to your suggestions @JTannerWMF - 8 is a lot, so removing (6) - Placeholder and (7) Image not available makes sense; as well as merging (4) and (5) to something with less jargon ("metadata") like I don't have enough information to know if this is a good match.

In a similar vein, maybe (3) could also be rewritten to be clearer what "our Wiki" is, since it is generally not so obvious to more casual users that there are different language Wikipedias. I wonder if the actual $language wiki name could be placed there instead, e.g., The image is not as relevant for Danish Wikipedia.

Finally, would there be an additional catch-all "Other/None of the above" option? If it is between that or (8) I didn't understand the task, my vote would be for the "Other" option instead.

JTannerWMF updated the task description. (Show Details)Tue, Feb 9, 8:05 PM

@JTannerWMF -- here are my thoughts on which options to include and how to phrase them. I took out the "on our Wiki" one because (a) I remember our ambassadors had a hard time understanding that one, so I imagine newcomers will, too (b) once our ambassadors were using it in their evaluations of the algorithm it basically never occurred. I took out 6 and 7 for the same reasons you did. And I added in one: "image is offensive", because we specifically wanted to learn how often this occurs so we can decide whether to prioritize adding an NSFW filter to the algorithm.

I wrote a "long version" for each option and a "short version". I wonder which one will look/feel right on the mobile screen.

Long version:

* Image doesn't show the subject of the article
* Image is of very low quality
* Not enough information to decide
* I don't know enough about the subject of the article
* Image is offensive
* Other

Short version:

* Not relevant
* Low quality
* Not enough information
* Don't know this subject
* Offensive
* Other
JTannerWMF updated the task description. (Show Details)Thu, Feb 11, 5:07 PM
schoenbaechler renamed this task from Design for image recommendations to Design for Add images usability tests.Wed, Feb 24, 12:25 PM
schoenbaechler updated the task description. (Show Details)
schoenbaechler moved this task from Doing to Done on the Android Design board.