This test is intended to help us understand what impact the Reply Tool is having on Junior Contributors' likelihood to start (activation) and continue (retention) participating on Wikipedia talk pages.
Decision to be made
The decision this analysis is intended to help us make:
Should the Reply tool be offered to all people, at all wikis, as an opt-out user preference?
Hypotheses
To help evaluate the impact of the Reply tool, we would like to analyze whether adding a more intuitive workflow for replying to specific comments to Wikipedia talk pages:
| ID | Hypothesis | Metric(s) for evaluation |
|---|---|---|
| KPI | ...causes a greater percentage of Junior Contributors to publish the comments they start without a significant increase in disruption. (see "Guardrail" below) | Comment completion rate as defined by the number of people who click the [ reply ] link (action = init), what % of people successfully publish the comment they were drafting (action = saveSuccess). |
| Guardrail | ...does not cause a significant increase in the number of disruptive edits being made to talk pages | The number of edits made to talk pages that are reverted within 48 hours. The number of editors who are blocked after making an edit to a talk page. |
| Curiosity #1 | ...causes a greater number of Junior Contributors to start participating productively on talk pages. | The number of distinct Junior Contributors who make at least one edit to a page in a talk namespace that is not reverted within 48 hours. |
| Curiosity #2 | ...causes a greater percentage of Junior Contributors continue participating productively on talk pages. | The percentage of Junior Contributors who who make at least one edit to a page in a talk namespace that is not reverted within 48 hours in each of the following time intervals: 2 to 7 days after making their edit (read: within the first week), 8 to 14 days after making their first edit (read: within the second week), and 15 to 30 days after making their first edit (read: within the third or fourth weeks). |
Decision matrix
| ID | Scenario | Plan of action |
|---|---|---|
| 1. | People are "meaningfully" more likely to publish edits using the Reply Tool than they are using full-page editing | Continue with plans to make the Reply Tool available at all Wikipedias, by default. See T269062 for more detail. |
| 2. | People are "meaningfully" less likely to publish edits using the Reply Tool than they are using full-page editing | Investigate where within the Reply Tool comment funnel people are dropping off and what could be contributing to this drop-off. In parallel, we will pause plans to make the Reply Tool available at all Wikipedias by default. |
| 3. | People are as likely to publish edits using the Reply Tool as they are using full-page editing | Continue with plans to offer features as opt-out preference at all Wikipedias considering we have meaningful qualitative feedback and quantitative data that suggests the tool is leading people to find participating on talk pages easier / more efficient.[ii] |
Open questions
- 1. Should edits to non-talk namespace pages be included in this analysis?
- 2. What wikis should be included in the test? See: T267379.
Done
- A report is published that evaluates the ===Hypotheses listed above
i. Editor experience buckets
- Logged out
- 0 cumulative edits
- 1-4 cumulative edits
- 5-99 cumulative edits
- 100-999 cumulative edits
- 1000+ cumulative edits
ii. An example of said "quantitative data": T247139