This analysis is intended to help us understand the impact Topic Subscriptions are having on the likelihood that contributors, across experience levels, will receive quicker responses to new topics they start and comments they post on wikitext talk pages.
Decision to be made
The decision this analysis is intended to help us make:
Should Topic Subscriptions be offered to all people, at all projects, as an opt-out user preference?
Analysis timing
TBD
Metrics
| ID | Description | Metric(s) for evaluation |
|---|---|---|
| KPI | Contributors, across experience levels, will receive quicker responses to new topics they start and comments they post on Wikitext talk pages because other volunteers will be made aware of these comments being posted and conversations being started in real-time. | For all comments and new topics with a response, the average time duration from a contributor posting on a talk page to them receiving a response from another person, grouped by experience level |
| Guardrail #1 | Topic Subscriptions should not cause a significant increase in disruptive behavior. | 1. Sharp increase in the number of notifications sent per day per contributor and 2. Percent of contributors that disable notifications |
| Guardrail #2 | Topic Subscriptions should not cause an increase in the amount of time it takes for people to receive responses to the conversations they start and comments they post | 1. Increase in the average time duration from a contributor posting on a talk page to receiving a response and 2. Increase in the percent of comments without a response after ___ (TBD) time. |
| Guardrail #3 | Topic Subscriptions should not cause a significant (read: sharp) increase or decrease in the number of Senior Contributors editing talk pages | Percent change in the number of Senior Contributors making edits to talk pages. |
| Curiosity #1 | Contributors, across experience levels, will post a greater number of comments to talk pages because they will be more aware of opportunities to offer support/guidance/expertise/etc. | 1. Average number of comments or new topics posted on talk pages by contributors that edit a talk page., 2. Percent of contributors that edit a talk page and start a new topic by number of topics (e.g. 1-5, 6-10, 11-15, etc), and 3. Percent of contributors by experience level that post at least one comment or one topic to a talk page. |
| Curiosity #2 | Contributors, across experience levels, will start a greater number of topics on talk pages because they will feel more confident the other person will respond. | Percent of contributors that edit a talk page and start a new topic grouped by the number of topics (e.g. 1-5, 6-10, 11-15, etc) they've started. |
| Curiosity #3 | The pace/cadence of conversations will change because the people who are interested in conversations will know more quickly when someone says something that motivates them to respond. | TBD |
Decision matrix
| ID | Scenario | Plan of action |
|---|---|---|
| 1. | People who have had access to topic subscriptions are "significantly" more likely to receive quicker responses to new topics they start and comments they post on wikitext talk pages than people who have not had access to topic subscriptions. | Continue with plans to make Topic Subscriptions available to all people, at all projects, by default. |
| 2. | People are "significantly" less likely to receive quicker responses to new topics they start and comments they post on wikitext talk pages | TBD |
| 3. | People who have had access to topic subscriptions are likely to receive a quicker response to the new topics they start and comments they post in the same amount of time as people who do not have access to topic subscriptions. | Barring any significant negative qualitative feedback about topic subscriptions, continue with plans to the feature to all people, at all projects, by default |
References
Done
- A report is published that evaluates the ===Metrics listed above
- A decision is made and documented about how the analysis's results will impact plans to offer topic subscriptions to more people at more projects.