This analysis is intended to help us understand the impact Topic Subscriptions are having on the likelihood that contributors, across experience levels, will respond more quickly to comments that are posted on wikitext talk pages.
Decision to be made
Should Topic Subscriptions be offered to all logged in volunteers, at all Wikimedia wikis, by default (read: as an opt-out user preference)?
A/B test timing
- Start date: Thursday, 2 June 2022
- End date: per conversation @MNeisler + @ppelberg had on 8 June, we will plan for the analysis to begin as early as Tuesday, 5 July 2022
Participating wikis
The list of wikis participating will be finalized once T304027 is resolved.
Hypotheses
To help evaluate the impact of Topic Subscriptions, we will analyze whether adding a way for being notified when new comments are added to discussions you have an expressed an interest in causes the following...
| ID | Hypothesis | Metric(s) for evaluation |
|---|---|---|
| KPI | Contributors, across experience levels, will respond more quickly to new topics and comments that are posted on Wikitext talk pages because they will be made aware of these new comments and topics in real-time. | For all comments and new topics with a response, the average time duration from "Person A" posting on a talk page and "Person B" posting a response, grouped by experience level of "Person A". Reason: Topic Subscriptions are designed to shorten the amount of time it takes for people to receive the input they are seeking. As such, we are measuring "waiting" time from the perspective of the person who is expecting a response, who in this scenario is "Person A." |
| Curiosity #1 | Contributors, across experience levels, will post a greater number of comments to talk pages because they will be more aware of opportunities to offer support/guidance/expertise/etc. | Average number of comments or new topics posted on talk pages by individual contributors that edit a talk page, grouped by experience level. Note: if time allows, we are also curious to see the total number of comments people within the two test groups post. |
| Curiosity #2 | Contributors, across experience levels, will start a greater number of topics on talk pages because they will feel more confident the other person will respond. | Percent of contributors that edit a talk page and start a new topic, grouped by the number of topics (e.g. 1-5, 6-10, 11-15, etc) they've started and experience level |
| Guardrail #1 | Topic Subscriptions should not cause a significant increase in disruptive behavior. | A) Sharp increase in the number of notifications sent/contributor/day and B) Sharp increase in the percent of contributors that disable notifications |
| Guardrail #2 | Topic Subscriptions should not cause a sharp increase in the amount of time it takes for people to respond to comments and new topics that are posted to wikitext talk pages. | Average time between when a new comment or new topic is published and someone responding to said comment or topic |
| Guardrail #3 | Topic Subscriptions should not cause a significant (read: sharp) increase or decrease in the number of Senior Contributors editing talk pages | Percent change in the number of Senior Contributors making edits to talk pages. |
Decision matrix
| ID | Scenario | Plan of action |
|---|---|---|
| 1. | People who have had access to topic subscriptions are "significantly" more likely to respond more quickly to new topics and comments that are posted on wikitext talk pages than people who have not had access to topic subscriptions. | Continue with plans to make Topic Subscriptions available to everyone who is logged in, at all projects, by default. |
| 2. | People are "significantly" less likely to respond more to new topics and comments that are posted on wikitext talk pages | Pause plans for wider deployment and prioritize an investigation in what could be contributing to the decreases in responsiveness |
| 3. | People who have had access to topic subscriptions are either "marginally" more or less likely to respond more quickly to new topics and comments that are posted on wikitext talk pages than people who do not have access to topic subscriptions. | Barring any significant negative qualitative feedback about topic subscriptions, continue with plans to the feature to everyone who is logged in, at all projects, by default |
Done
- A report is published that evaluates the ===Metrics listed above
- A decision is made and documented about how the analysis's results will impact plans to offer topic subscriptions to more people at more projects.
References
Notes: In T290508, we decided that as part of this A/B test of Topic Subscriptions, we will look at whether there has been a stark decrease in the percentage of main namespace edits as a proxy for whether Topic Subscriptions are decreasing peoples' engagement with their Watchlists and thus, causing them to be less likely to notice and undo vandalism.
