This analysis is intended to help us understand the impact Topic Subscriptions are having on the likelihood that contributors, across experience levels, will receive quicker responses to new topics they start and comments they post on wikitext talk pages.
=== Decision to be made
**Should Topic Subscriptions be offered to all logged in volunteers, at all Wikimedia wikis, by default (read: as an opt-out user preference)?**
=== A/B test timing
- Start date: **TBD date //after// T290517 is resolved**
- End date: **per conversation @mneisler + @ppelberg had on 13 April, we will plan for the analysis to begin ~2 weeks after T304029 is resolved.**
=== Participating wikis
//The list of wikis participating will be finalized once T304027 is resolved.//
=== Hypotheses
//To help evaluate the impact of Topic Subscriptions, we will analyze whether adding a way for being notified when new comments are added to discussions you have an expressed an interest in causes the following...//
|ID|Hypothesis|Metric(s) for evaluation
|---|---|---
|**KPI**|Contributors, across experience levels, will receive quicker responses to new topics they start and comments they post on Wikitext talk pages because other volunteers will be made aware of these comments being posted and conversations being started in real-time.| For all comments and new topics with a response, the average time duration from a contributor posting on a talk page to them receiving a response from another person, grouped by experience level
|**Curiosity #1**|Contributors, across experience levels, will post a greater number of comments to talk pages because they will be more aware of opportunities to offer support/guidance/expertise/etc. | Average number of comments or new topics posted on talk pages by contributors that edit a talk page, grouped by experience level
|**Curiosity #2**|Contributors, across experience levels, will start a greater number of topics on talk pages because they will feel more confident the other person will respond.|Percent of contributors that edit a talk page and start a new topic, grouped by the number of topics (e.g. 1-5, 6-10, 11-15, etc) they've started and experience level
|**Guardrail #1**|Topic Subscriptions should not cause a significant increase in disruptive behavior.|A) Sharp increase in the number of notifications sent/contributor/day and B) Sharp increase in the percent of contributors that disable notifications
|**Guardrail #2**|Topic Subscriptions should not cause a sharp increase in the amount of time it takes for people to receive responses to the conversations they start and comments they post |Average time between a contributor posting a new comment or new topic on a talk page and them receiving a response
|**Guardrail #3**|Topic Subscriptions should not cause a significant (read: sharp) increase or decrease in the number of Senior Contributors editing talk pages|Percent change in the number of Senior Contributors making edits to talk pages.
=== Decision matrix
|ID|Scenario|Plan of action
|---|---|---
|1.|People who have had access to topic subscriptions are "significantly" **more likely** to receive quicker responses to new topics they start and comments they post on wikitext talk pages than people who have //not// had access to topic subscriptions. | Continue with plans to make Topic Subscriptions available to everyone who is logged in, at all projects, by default.
|2.|People are "significantly" **less likely** to receive quicker responses to new topics they start and comments they post on wikitext talk pages |Pause plans for wider deployment and prioritize an investigation in what could be contributing to the decreases in responsiveness
|3.|People who have had access to topic subscriptions are either "marginally" more or less likely to receive a quicker response to the new topics they start and comments they post than people who do not have access to topic subscriptions.|Barring any significant negative qualitative feedback about topic subscriptions, continue with plans to the feature to everyone who is logged in, at all projects, by default
=== Done
- [ ] A report is published that evaluates the `===Metrics` listed above
- [ ] A decision is made and documented about how the analysis's results will impact plans to offer topic subscriptions to more people at more projects.
---
=== References
- [Talk pages project/Notifications/Measurement plan](https://docs.google.com/document/d/15eD960rWRzxqYNQpjgsqd8nGwW9cRrpzOfYVd7nkX-0/edit#)
---
//Notes: In T290508, we decided that as part of this A/B test of Topic Subscriptions, we will look at whether there has been a stark decrease in the percentage of main namespace edits as a proxy for whether Topic Subscriptions are decreasing peoples' engagement with their Watchlists and thus, causing them to be less likely to notice and undo vandalism.//