Page MenuHomePhabricator
Create Task
Maniphest T292316

Analytics for image suggestions notifications
Closed, ResolvedPublic
Actions

Description

As a product manager for Structured Data, I want to understand how the image suggestions notifications feature is being used and how it relates to other notifications behavior, so that I can make decisions about how to build and/or make changes to the feature going forward, and determine its success.

Ideally we can create a dashboard with this information.

The high level questions we want to answer are:

  1. Is our communication about our suggestions effective?
  2. Do Editors include adding suggested images in their current behavior flow?
  3. Are the suggestions themselves good (this tickets informs how do we define good)?

Specific analytics questions to answer/acceptance criteria:

  • How many notifications does a user get today? (We are starting with 2 notifications max per week; but this information will help us decide if we should increase or decrease that number in the future). (Covered in T291403)
  • How many articles is a user typically watching? (We may increase the number of notifications in the future based on this number; and may also let us know if we need a landing page or tool where users can interact with suggestions in the future if users would see lots of notifications/suggestions). (Covered in T291403)
  • How many images on average are added by a user per month? (To understand what the current behavior is today and how people engage with images; would help us know if engagement increases based on our work). (Covered in T299667)
  • Number of notifications sent (To get a sense of the scope of our work and whether we should increase or decrease) (should be queryable from the Echo tables, @cchen to confirm)
  • Number of opt-outs (To see whether users are annoyed by our notifications or not interested in them, etc) (Covered in T292146)
  • Number of images suggested that are added to the matched article within a month of receiving the notification (To see whether the notifications are successful) (Covered in T299667)
  • Number of suggested images not reverted from their matched article (low revert rate to show that the matches are good) (Covered in T299667)

Related Objects
Search...

Event Timeline

CBogen created this task.Oct 1 2021, 7:39 PM
CBogen added a parent task: T292142: [EPIC] Image Suggestions Notifications for More Experienced Contributors.
CBogen mentioned this in T291578: Define what user behavior we want to track for image suggestions.Oct 1 2021, 7:43 PM
ldelench_wmf moved this task from Triage to Current Quarter on the Product-Analytics board.Oct 5 2021, 5:13 PM
CBogen updated the task description. (Show Details)Oct 7 2021, 7:04 PM
cchen mentioned this in T292785: Instrumentation for image suggestions notifications.Oct 15 2021, 6:07 AM
cchen triaged this task as Medium priority.Oct 25 2021, 4:35 PM
ldelench_wmf assigned this task to cchen.Nov 16 2021, 6:15 PM
cchen moved this task from Current Quarter to Upcoming Quarter on the Product-Analytics board.Nov 22 2021, 5:14 PM
Sneha moved this task from Triage to Current Work on the Structured-Data-Backlog board.Jan 10 2022, 5:32 PM
Sneha edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.
CBogen edited projects, added Structured-Data-Backlog; removed Structured-Data-Backlog (Current Work).Jan 10 2022, 5:36 PM
CBogen added a subtask: T299667: Track whether image suggestions notifications lead to media additions.Jan 20 2022, 6:29 PM
CBogen updated the task description. (Show Details)Jan 21 2022, 2:05 PM
CBogen updated the task description. (Show Details)Jan 21 2022, 2:21 PM
cchen moved this task from Upcoming Quarter to Current Quarter on the Product-Analytics board.Jan 24 2022, 5:23 PM
CBogen moved this task from Triage to Analytics on the Structured-Data-Backlog board.Jan 24 2022, 5:34 PM
CBogen mentioned this in T300065: Determine how many images on average are added by a user per month.Jan 25 2022, 5:34 PM
CBogen updated the task description. (Show Details)Jan 25 2022, 7:22 PM
Seddon changed the status of subtask T292785: Instrumentation for image suggestions notifications from Open to In Progress.Feb 21 2022, 3:57 PM
cchen closed subtask T292785: Instrumentation for image suggestions notifications as Resolved.Mar 7 2022, 11:00 PM
cchen moved this task from Current Quarter to Upcoming Quarter on the Product-Analytics board.May 17 2022, 5:18 PM
CBogen updated the task description. (Show Details)Jul 1 2022, 1:09 PM
CBogen moved this task from To Do to SD: Experienced Contributor Notifications on the Image-Suggestions board.Jul 7 2022, 6:39 PM
CBogen mentioned this in T300706: Establish baseline for media additions.Jul 14 2022, 2:16 PM
cchen moved this task from Upcoming Quarter to Kanban on the Product-Analytics board.Oct 5 2022, 6:22 AM
cchen edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
cchen moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.
cchen added a comment.Oct 5 2022, 5:22 PM

A first pass on the analysis of image suggestion notifications can be found in this Jupyter notebook. The data is from 07/20/2022 - 08/31/2022.

For quick reference, the results are as follows:

Number of notifications sent

wiki_dbnotification_sentnotification_readread_pct
idwiki5913167428.31%
ptwiki18988432922.80%
ruwiki452641337629.55%

Number of opt-outs
For image suggestion notification, by default, we set web notification and push notification on, and email notification off. To look for opt_outs we will take a look at how many users turn off echo-subscriptions-<type>-image-suggestions for web and app.
Opt-out Rate = Number of opt-outs / Number of users who received image suggestion notifications

wiki_dbtypeoptout_userstotal_usersopt_out_pct
idwikipush98021.12%
idwikiweb108021.25%
ptwikipush2025160.79%
ptwikiweb2325160.91%
ruwikipush10660101.76%
ruwikiweb11260101.86%

Number of Images Added by Users in August

  • Russian Wikipedia:
    • Average number of images added by each user: 3
    • Among all the editors and edits: 5.7% of edits are image edits, 26 % of editors have image edits, 22,005 images are added.
  • Portuguese Wikipedia:
    • Average number of images added by each user: 2
    • Among all the editors and edits: 7.5% of edits are image edits, 22.2 % of editors have image edits, 9,264 images are added.
  • Indonesian Wikipedia:
    • Average number of images added by each user: 3
    • Among all the editors and edits: 8.0% of edits are image edits, 29.3 % of editors have image edits, 7,203 images are added.

Number of Images Added by Experienced Users in August

  • Russian Wikipedia:
    • Average number of images added by each user: 8
    • Among all the editors and edits: 5.2% of edits are image edits, 45.9 % of editors have image edits, 17,895 images are added.
    • Number of images added to unillustrated articles: 4,529
  • Portuguese Wikipedia:
    • Average number of images added by each user: 7
    • Among all the editors and edits: 6.7% of edits are image edits, 49.4% of editors have image edits, 6,318 images are added.
    • Number of images added to unillustrated articles: 1,846
  • Indonesian Wikipedia:
    • Average number of images added by each user: 12
    • Among all the editors and edits: 7.1% of edits are image edits, 59.6 % of editors have image edits, 4,619 images are added.
    • Number of images added to unillustrated articles: 1,416

Number of Images Suggested that are Added to the Matched Article
To find the image edits to the matched articles, we are looking for image edits timestamp after notification read timestamp. We also want to exclude image edits from Newcomer Tasks.
There are in total 505 suggested images added to matched articles.

wiki_dbimages_addedrelated_user
idwiki5637
ptwiki12579
ruwiki324208

Number of suggested images not reverted from their matched article
The revert rates of suggested image edits are 0% in all three wikis, given the overall revert rate of image edits by experienced users is ~2%.

mpopov subscribed.Feb 1 2023, 5:28 PM

@cchen: What's the status with this? Is this done?

@CBogen: Did you and/or Alexandra use the data that Connie provided back in October to make any decisions?

CBogen added a comment.Feb 1 2023, 5:31 PM
In T292316#8579215, @mpopov wrote:

@cchen: What's the status with this? Is this done?

@CBogen: Did you and/or Alexandra use the data that Connie provided back in October to make any decisions?

Hi @mpopov -- yes, the data has been critical in reporting on the SDAW grant and Alexandra will be using it to decide on next steps for the product after the grant period.

However, this isn't done, because we are still waiting for the dashboard in T299667: Track whether image suggestions notifications lead to media additions. The last update from @cchen was a few weeks ago, when she said that the test dashboard was in process and that we should have a dashboard by end of Jan. @cchen do you have an update on that?

mpopov added a comment.Feb 1 2023, 5:40 PM

@CBogen: Thank you for the quick response! That is very helpful to know.

mpopov added a subscriber: AUgolnikova-WMF.Feb 1 2023, 7:56 PM
CBogen closed subtask T299667: Track whether image suggestions notifications lead to media additions as Resolved.Mar 9 2023, 3:35 PM
CBogen closed this task as Resolved.Mar 9 2023, 6:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL