This test is intended to help us understand what impact the Reply Tool is having on Junior Contributors' likelihood to start (activation) and continue (retention) participating on Wikipedia talk pages.
===Decision to be made
//The decision this analysis is intended to help us make://
**Should the Reply tool be offered to all people, at all wikis, as an //opt-out// user preference? **
=== Hypotheses
//To help evaluate the impact of the Reply tool, we would like to analyze whether adding a more intuitive workflow for replying to specific comments to Wikipedia talk pages://
|ID|Hypothesis|Metric(s) for evaluation|
|---|---|---
|KPI| **...causes** a greater percentage of Junior Contributors to publish the comments they start //without// a significant increase in disruption. (see "Guardrail" below)| Comment completion rate as defined by the number of people who click the `[ reply ]` link (action = `init`), what % of people successfully publish the comment they were drafting (action = `saveSuccess`).
|Guardrail| **...does not cause** a significant increase in the number of disruptive edits being made to talk pages| The number of edits made to talk pages that are reverted within 48 hours. The number of editors who are blocked after making an edit to a talk page.
|Curiosity #1| **...causes** a greater number of Junior Contributors to //start// participating productively on talk pages.| The number of distinct Junior Contributors who make at least one edit to a page in a talk namespace that is not reverted within 48 hours. |
|Curiosity #2| **...causes** a greater percentage of Junior Contributors //continue// participating productively on talk pages.| The percentage of Junior Contributors who who make at least one edit to a page in a talk namespace that is not reverted within 48 hours in each of the following time intervals: **2 to 7 days after making their edit** (read: within the first week), **8 to 14 days after making their first edit** (read: within the second week), and **15 to 30 days after making their first edit** (read: within the third or fourth weeks).
=== Decision matrix
|ID|Scenario|Plan of action
|---|---|---
|1.|People are "meaningfully" **more likely** to publish edits using the Reply Tool than they are using full-page editing | Continue with plans to make the Reply Tool available at all Wikipedias, by default. //See T262331 for more detail.//
|2.|People are "meaningfully" **less likely** to publish edits using the Reply Tool than they are using full-page editing |Investigate where within the Reply Tool comment funnel people are dropping off and what could be contributing to this drop-off. In parallel, we will pause plans to make the Reply Tool available at all Wikipedias by default.
|3.|People are **as likely** to publish edits using the Reply Tool as they are using full-page editing|Continue with plans to offer features as opt-out preference at all Wikipedias considering we have meaningful qualitative feedback and quantitative data that suggests the tool is leading people to find participating on talk pages easier / more efficient.[ii]
=== Open questions
- [ ] 1. Should edits to non-talk namespace pages be included in this analysis?
- [ ] 2. What wikis should be included in the test? //See: T267379.//
=== Done
- [ ] A report is published that evaluates the `===Hypotheses` listed above
---
i. Editor experience buckets
- Logged out
- 0 cumulative edits
- 1-4 cumulative edits
- 5-99 cumulative edits
- 100-999 cumulative edits
- 1000+ cumulative edits
ii. An example of said "quantitative data": T247139