This task is about conducting an A/B test to help us understand what impact the New Discussion Tool is having on Junior Contributors' likelihood to start (activation) and continue (retention) participating on Wikipedia talk pages.
=== Test timing
- Start date: **TBD**
- End date: **~4 weeks after "Start date"** per T290204#7356556
=== Participating wikis
//The list of participating wikis will be finalized in T291306.//
=== Decision to be made
//The decision this analysis is intended to help us make://
**Should the New Discussion Tool be offered to all people, at all wikis, as an opt-out user preference?**
//To help evaluate the impact of the New Discussion Tool, we will analyze whether adding a more intuitive workflow for starting new discussions on Wikipedia talk pages://
|ID|Hypothesis|Metric(s) for evaluation
|KPI|**...causes** a greater percentage of Junior Contributors to publish the new discussions they start without a significant increase in disruption. (see "Guardrail" below) | New discussion completion rate as defined by the number of people who click the `Add topic` / `New section` link (action = `init`), what % of people successfully publish the discussion topics they were drafting (action = `saveSuccess`).
| Guardrail|**...does not cause** a significant increase in the number of disruptive edits being made to talk pages|The number of edits made to talk pages that are reverted within 48 hours. The number of editors who are blocked after making an edit to a talk page.
|Curiosity #1|**...causes** a greater percentage of Junior Contributors to //start// participating productively on talk pages.| The number of distinct Junior Contributors who make at least one edit to a page in a talk namespace that is not reverted within 48 hours.
|Curiosity #2|**...causes** a greater percentage of Junior Contributors continue participating productively on talk pages.| The percentage of Junior Contributors who who make at least one edit to a page in a talk namespace that is not reverted within 48 hours in each of the following time intervals: 2 to 7 days after making their edit (read: within the first week), 8 to 14 days after making their first edit (read: within the second week), and 15 to 30 days after making their first edit (read: within the third or fourth weeks).
=== Decision matrix
|ID|Scenario|Plan of action
|1.|People are **more likely** to publish edits using the New Discussion Tool than they are using the existing `section=new` interface|Continue with plans to make the New Discussion Tool available at all Wikipedias, by default. //See T275256 for more detail.//
|2.|People are **less likely** to publish edits using the New Discussion Tool than they are using the existing `section=new` workflow.|Investigate where within the New Discussion Tool workflow people are dropping off and hypothesize what could be contributing to this drop-off. In parallel, we will pause plans to make the New Discussion Tool available at all Wikipedias, by default. //See: T263054#7363191.//
|3.|People are **as likely** to publish edits using the New Discussion Tool as they are using the existing `section=new` workflow.|Continue with plans to offer the New Discussion Tool as an opt-out preference at all Wikipedias considering qualitative feedback suggests the tool is leading people to find participating on talk pages easier / more efficient.
- [ ] Finalize `===Hypotheses` and `===Decision matrix`
- [ ] Complete analysis
- [ ] Publish findings on mediawiki.org: [Talk pages project/New discussion](https://www.mediawiki.org/wiki/Talk_pages_project/New_discussion)