Page MenuHomePhabricator

Evaluate impact of notification interventions
Open, MediumPublic

Description

This analysis is intended to help us understand the impact Topic Subscriptions are having on the likelihood that contributors, across experience levels, will receive quicker responses to new topics they start and comments they post on wikitext talk pages.

Decision to be made

The decision this analysis is intended to help us make:
Should Topic Subscriptions be offered to all people, at all projects, as an opt-out user preference?

Analysis timing

TBD

Metrics

IDDescriptionMetric(s) for evaluation
KPIContributors, across experience levels, will receive quicker responses to new topics they start and comments they post on Wikitext talk pages because other volunteers will be made aware of these comments being posted and conversations being started in real-time.For all comments and new topics with a response, the average time duration from a contributor posting on a talk page to them receiving a response from another person, grouped by experience level
Guardrail #1Topic Subscriptions should not cause a significant increase in disruptive behavior.1. Sharp increase in the number of notifications sent per day per contributor and 2. Percent of contributors that disable notifications
Guardrail #2Topic Subscriptions should not cause an increase in the amount of time it takes for people to receive responses to the conversations they start and comments they post1. Increase in the average time duration from a contributor posting on a talk page to receiving a response and 2. Increase in the percent of comments without a response after ___ (TBD) time.
Guardrail #3Topic Subscriptions should not cause a significant (read: sharp) increase or decrease in the number of Senior Contributors editing talk pagesPercent change in the number of Senior Contributors making edits to talk pages.
Curiosity #1Contributors, across experience levels, will post a greater number of comments to talk pages because they will be more aware of opportunities to offer support/guidance/expertise/etc.1. Average number of comments or new topics posted on talk pages by contributors that edit a talk page., 2. Percent of contributors that edit a talk page and start a new topic by number of topics (e.g. 1-5, 6-10, 11-15, etc), and 3. Percent of contributors by experience level that post at least one comment or one topic to a talk page.
Curiosity #2Contributors, across experience levels, will start a greater number of topics on talk pages because they will feel more confident the other person will respond.Percent of contributors that edit a talk page and start a new topic grouped by the number of topics (e.g. 1-5, 6-10, 11-15, etc) they've started.
Curiosity #3The pace/cadence of conversations will change because the people who are interested in conversations will know more quickly when someone says something that motivates them to respond.TBD

Decision matrix

IDScenarioPlan of action
1.People who have had access to topic subscriptions are "significantly" more likely to receive quicker responses to new topics they start and comments they post on wikitext talk pages than people who have not had access to topic subscriptions.Continue with plans to make Topic Subscriptions available to all people, at all projects, by default.
2.People are "significantly" less likely to receive quicker responses to new topics they start and comments they post on wikitext talk pagesTBD
3.People who have had access to topic subscriptions are likely to receive a quicker response to the new topics they start and comments they post in the same amount of time as people who do not have access to topic subscriptions.Barring any significant negative qualitative feedback about topic subscriptions, continue with plans to the feature to all people, at all projects, by default

References

Done

  • A report is published that evaluates the ===Metrics listed above
  • A decision is made and documented about how the analysis's results will impact plans to offer topic subscriptions to more people at more projects.

Event Timeline

ppelberg updated the task description. (Show Details)
ppelberg edited projects, added Editing-team (Tracking); removed Editing-team.
ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.
MNeisler triaged this task as Medium priority.May 24 2021, 1:10 PM