Notes from conversations

2020-12-07 | Dmitry Brant

• Miriam is continuing to go through the updates.
• Mixed messages from services team re: timelines when it would be ready (something ready or not ready in Q3).
• Service might not be available in Q3.
• Where do we insert the image in the article that get suggested?
  • DB: Suggests to insert image at the top
  • If we always insert it at the top, the community is fine with it
    • Feature parity with visual editor
  • Image in articles have caption
    • Is the user also supposed to add that caption
    • Adding caption is the next flow

Notes from conversations

2020-12-07 | Dmitry Brant

• DB: Miriam created a dump spreadsheet that we can use to create a prototype.
• DB and RS agree on the direction → map and collect ideas for Android, then roughly sketch them out until the end of this week (or end of year since RS is working in the week of Dec 28)

Notes from conversations

2020-12-16 | Pau Giner

• PG encourages to think about how to connect different tasks to keep people engaged.
• PG advises to map/draw different directions for IR on Android
  • Compile behavior and ideas of a new project
  • His process is to map items into motivation, discovery, etc., here’s an example Trello board for multilingual editors
  • RS thinks about an exercise to create a mindmap and then group / sort ideas around it.
• RS mentions cross collaboration with Jess to talk about image insertion in visual editor
• PG: Research in this are? Learn from people that are adding images (Rita)

Notes from conversations

2021-01-13 | Jazmin Tanner

• Explore interaction model of dating apps (swipe left and right?)
• How do we deal with languages?
  • Do we support any other language than English Wiki?
  • Do we want to showcase images that have a caption in other (app) languages?
  • Options to choose language?
  • Caption language?
• Onboarding to image recommendations?
• Survey for image recommendations? How do we display the survey

Notes from conversations

2021-01-19 | Rita Ho

• The aim of the Suggested edits feed experience is to help improve the algorhithm
• Is it then a demo?
  • “Algorhithm Gym”
  • Not disappointed and confused by not making an edit.
  • Separate version of adding images.
  • Positive reinforcements since users don’t edit Wikipedia (Trainer like crowdsource).
  • Contributions are not counted.
• Data
  • People looked at image info a lot in the usablity tests.
  • Difficulty of adressing newcomers (Wikipedia vs Commons)
  • Find out if the feature is for more advanced editors on Android.
• Usability test, two variants
  • Optional captions were part of the test.
  • After they did 20 imagre recommendations they had to add few (optional) caption in the end.
  • Captions were really bad.
    • Do we want to include these?

Notes from conversations

2021-01-19 | Chat with Growth

• RS: Can we find out which meta data is the most important?
• RH: Convey algorithm gym task aspect
• MM: Stats or credit on progress of the task. Positive reinforcements.
• RH: Prioritize infos
• MM: Convey that image is not there yet. RS will explore directions for this.
• RH: Audience is bigger on Android. A lot of the metadata is written in English. Figure out how to deal with languages.
• RH: Consider diary study from Carolyn on iOS analytics data (“Appsy”).
• Additional links from Rita (Jan 20) per Slack message:
  • Android Explore Feed usage
  • Apps reading list usage
  • Android + iOS User testing participants recruitment survey - demographic and basic usage results from a mobile app test participants recruiting drive.
  • iOS longitudinal Diary study (involved a serving users a series of short surveys over time, as well as capturing analytics usage via a third vendor AppSee)

@Dbrant @JTannerWMF

Per our discussion from Wednesday, I outlined a few possible experiments on positive reinforcements for the “Add images” task. The suggestions base on a broader research piece.

👉Check out chapter 3.4. in the task’s description and please review it / leave comments before our next sync meeting.

@RHo had a chat about elements of positive reinforcement yesterday. Adding notes from our conversation:

• RH suggests a lightweight element (notifications / thanks / message), that is contained to the “Add images” task to avoid overhead
  • It’d probably make sense to build on notifications, since they proved to work
    • Put effort into crafting the message (RS, note to self)
    • Are the old ones still in place? (RS, needs to check)
    • Try out different notification messaging with A/B testing (RS)
• RH has the feeling that we’re pinning to much on to image recommendations
  • The main purpose of the task is to train the tool
  • How would we test if a specific element of positive reinforcement is successful? A/B test with badges? Does it give an answer if the specific positive reinforcement tactics work or not?
• About a notification when the new feature is available (new sell)
  • RH and RS are both on the same page that it might feel too intrusive.
• RH: Is continuous usage of the tool important? As it’s a prototype to train the algorithm?

@RHo, feel free to unsubscribe from the task again after reading this. I know, too many notifications are unhealthy (ironically to the topic of our conversation yesterday 🙃)

Thanks @schoenbaechler for capturing notes here. I added a few additional notes below but this all sounds right.

In T269594#6783415, @schoenbaechler wrote:

@RHo had a chat about elements of positive reinforcement yesterday. Adding notes from our conversation:

• RH suggests a lightweight element (notifications / thanks / message), that is contained to the “Add images” task to avoid overhead

Potentially just a count of number of images reviewed within the "Add images" task if we want people to track how many they've done, but making sure it is separate from the main "Contributions" section.

• • It’d probably make sense to build on [notifications (https://docs.google.com/presentation/d/1L6DOn6nY8aTR0GgJNSYML4E4H4v0Za1E6ji3WsA7X9M/edit#slide=id.g8b15d94b3a_0_478), since they proved to work
    • Put effort into crafting the message (RS, note to self)
    • Are the old ones still in place? (RS, needs to check)
    • Try out different notification messaging with A/B testing (RS)
• RH has the feeling that we’re pinning too much on to image recommendations
  • The main purpose of the task is to train the tool
  • How would we test if a specific element of positive reinforcement is successful? A/B test with badges? Does it give an answer if the specific positive reinforcement tactics work or not?

Yes, these were really rhetorical questions as I think it would be diluting the focus if positive reinforcement tactics are also being tested at the same time as this initial image matching MVP.

• About a notification when the new feature is available (new sell)
  • RH and RS are both on the same page that it might feel too intrusive.
• RH: Is continuous usage of the tool important? As it’s a prototype to train the algorithm?

Along the same lines as above, my concern is that by providing specific incentives like awards/badges for working on something that is a temporary contribution type not resulting in an edit, it may be a confusing message to send, esp. for something that located on the "Edits" part of the app.

---

@RHo, feel free to unsubscribe from the task again after reading this. I know, too many notifications are unhealthy (ironically to the topic of our conversation yesterday 🙃)

Thanks @schoenbaechler - all good!

Hey @schoenbaechler a thought I had for social connection is to show what percentile the person is in the for the day as it relates to task completion as opposed to how many people have contributed. Not sure if that is technically possible @Dbrant ?

Notes from the conversation with @JTannerWMF earlier today:

• RS and JT are going to create a slide deck for the “Image Recommendations MVP Sync” meeting on Wed, Feb 3, 2021
• JT mentions to go through the questions T273055 and add decisions / recommendations to it
• JT is going to create a project page for image recommendation on Android and RS will add thoughts from a design perspective to it.
• Copy suggestions for description in the main screen
  • Help us understand how well these images match with Wikipedia articles.
  • Help us improve the image matching algorithm.
  • Help to improve image matching suggestions for articles.
  • Help us train artificial intelligence to suggest matching images to articles.
• RS mentions UX writing for machine-assisted editing deck, which is interesting to study for crafting the copy for the task. JT will have a look at it.
• Onboarding
  • Both agree to make the onboarding dialog’s text more succinct.
  • JT suggests tutorial onboarding, RS recommends contextual onboarding with improved tooltips (that include “GOT IT” to dismiss them)
• RS will create visual for JT’s “Tinder” swipe right (yes) and left (no) idea
• JT and RS will define what happens when users taps, “Yes”, “No” and “Not sure” in detail in the deck

Notes from the recent conversation with @Dbrant:

• DB: Consider to make the image larger (full width)
• DB: Provides more info in regards to “Not sure”, “No”, ”Yes” options
  • If users press “Yes”, we pass the data to improve algorithm.
  • The “No” or “Not sure” dropdown should provide reasons for not being able to complete the task:
    • Image is plainly wrong
    • Subtly wrong (slight difference)
    • Caption in the wrong language (“I don’t understand the caption”)
    • Maybe one or two other choices (tbd)
  • DB mentions that the most simple pattern to present these options is a modal dialog dialog
    • RS shows options to make this more in context, e.g. by presenting message options to reply
  • Chat about positive reinforcement concepts in the task. DB’s feedback:
    • “X people contributed“ is not possible (we don’t have that number)
      • There are maybe possibilities for it but not for the first iteration (needs server side technology and can be done in a future quarter)
    • Levels
      • Need server side technology as well
    • Award
      • Adding an award to yourself is feels unnatural to Wikipedia
      • Barnstars are added by others
      • External account should award you for the edits

In T269594#6793296, @schoenbaechler wrote:

Notes from the recent conversation with @Dbrant:

• DB: Consider to make the image larger (full width)
• DB: Provides more info in regards to “Not sure”, “No”, ”Yes” options
  • If users press “Yes”, we pass the data to improve algorithm.
  • The “No” or “Not sure” dropdown should provide reasons for not being able to complete the task:
    • Image is plainly wrong
    • Subtly wrong (slight difference)
    • Caption in the wrong language (“I don’t understand the caption”)
    • Maybe one or two other choices (tbd)
  • DB mentions that the most simple pattern to present these options is a modal dialog dialog
    • RS shows options to make this more in context, e.g. by presenting message options to reply

Hi @schoenbaechler - some more relevant info on the "No reasons" selection.
On the "Add a link" structured task, we are showing a dialog with multiple options relevant to why a user would reject the suggestion as per DB's note.
You can see the task here: https://phabricator.wikimedia.org/T269647
Notes:

• While the reasons for the image not matching would be different from links, I would recommend to always provide an "Other" option.
• We are making the rejection reason optional, but I think you could make it mandatory so long as there is an "Other" option available
• The user should be able to change their reason after selection (in case of wrong tap, etc) so the dialog should be able to be brought back after selection.

The root ticket T268704 has a links to a mobile and desktop prototype in case you'd like to play with the whole Add links flow.

• Chat about positive reinforcement concepts in the task. DB’s feedback:
  • “X people contributed“ is not possible (we don’t have that number)
    • There are maybe possibilities for it but not for the first iteration (needs server side technology and can be done in a future quarter)
  • Levels
    • Need server side technology as well
  • Award
    • Adding an award to yourself is feels unnatural to Wikipedia
    • Barnstars are added by others
    • External account should award you for the edits

It does seems like the most apt and feasible positive reinforcement option is to show a count of image suggestions evaluated by the user. I do think these award and recognition types we discussed in the Positive Reinforcements research as being more appropriate to consider for more "consequential" actual article edits down the track.

From @Miriam :

The image is not depicting the subject of the article
The image is of very low quality
The image is relevant, but we would not use it on our Wiki
There is not enough metadata to understand the goodness of the image match
I am not expert enough on the article subject to judge the goodness of the image match
The image is just a placeholder for missing images
The image is not available
I did not understand the task

If you think 8 options are too many for this use case, we could skip/merge number (6) and (7), as they are just a special case for (1), and maybe merge (4) and (5).
Please let me know your thoughts, I am happy to iterate over those as much as needed.

In T269594#6815568, @JTannerWMF wrote:

From @Miriam :

The image is not depicting the subject of the article
The image is of very low quality
The image is relevant, but we would not use it on our Wiki
There is not enough metadata to understand the goodness of the image match
I am not expert enough on the article subject to judge the goodness of the image match
The image is just a placeholder for missing images
The image is not available
I did not understand the task

If you think 8 options are too many for this use case, we could skip/merge number (6) and (7), as they are just a special case for (1), and maybe merge (4) and (5).
Please let me know your thoughts, I am happy to iterate over those as much as needed.

+1 to your suggestions @JTannerWMF - 8 is a lot, so removing (6) - Placeholder and (7) Image not available makes sense; as well as merging (4) and (5) to something with less jargon ("metadata") like I don't have enough information to know if this is a good match.

In a similar vein, maybe (3) could also be rewritten to be clearer what "our Wiki" is, since it is generally not so obvious to more casual users that there are different language Wikipedias. I wonder if the actual $language wiki name could be placed there instead, e.g., The image is not as relevant for Danish Wikipedia.

Finally, would there be an additional catch-all "Other/None of the above" option? If it is between that or (8) I didn't understand the task, my vote would be for the "Other" option instead.

@JTannerWMF -- here are my thoughts on which options to include and how to phrase them. I took out the "on our Wiki" one because (a) I remember our ambassadors had a hard time understanding that one, so I imagine newcomers will, too (b) once our ambassadors were using it in their evaluations of the algorithm it basically never occurred. I took out 6 and 7 for the same reasons you did. And I added in one: "image is offensive", because we specifically wanted to learn how often this occurs so we can decide whether to prioritize adding an NSFW filter to the algorithm.

I wrote a "long version" for each option and a "short version". I wonder which one will look/feel right on the mobile screen.

Long version:

* Image doesn't show the subject of the article
* Image is of very low quality
* Not enough information to decide
* I don't know enough about the subject of the article
* Image is offensive
* Other

Short version:

* Not relevant
* Low quality
* Not enough information
* Don't know this subject
* Offensive
* Other