This task is about conducting an A/B test to help us understand what impact the New Discussion Tool is having on Junior Contributors' likelihood to start (activation) and continue (retention) participating on Wikipedia talk pages.
Test timing
- Start date: TBD
- End date: ~4 weeks after "Start date" per T290204#7356556
Participating wikis
This list of participating wikis is drawn from T291306.
Candidate wikis
| ID | Wiki | Code |
|---|---|---|
| 1. | Amharic Wikipedia | amwiki |
| 2. | Bengali Wikipedia | bnwiki |
| 3. | Chinese Wikipedia | zhwiki |
| 4. | Dutch Wikipedia | nlwiki |
| 5. | Egyptian Wikipedia | arzwiki |
| 6. | French Wikipedia | frwiki |
| 7. | Hebrew Wikipedia | hewiki |
| 8. | Hindi Wikipedia | hiwiki |
| 9. | Indonesia Wikipedia | idwiki |
| 10. | Italian Wikipedia | itwiki |
| 11. | Japanese Wikipedia | jawiki |
| 12. | Korean Wikipedia | kowiki |
| 13. | Oromo Wikipedia | omwiki |
| 14. | Persian Wikipedia | fawiki |
| 15. | Polish Wikipedia | plwiki |
| 16. | Portuguese Wikipedia | ptwiki |
| 17. | Spanish Wikipedia | eswiki |
| 18. | Thai Wikipedia | thwiki |
| 19. | Ukrainian Wikipedia | ukwiki |
| 20. | Vietnamese Wikipedia | viwiki |
Decision to be made
The decision this analysis is intended to help us make:
Should the New Discussion Tool be offered to all people, at all wikis, as an opt-out user preference?
Hypotheses
To help evaluate the impact of the New Discussion Tool, we will analyze whether adding a more intuitive workflow for starting new discussions on Wikipedia talk pages:
| ID | Hypothesis | Metric(s) for evaluation |
|---|---|---|
| KPI | ...causes a greater percentage of Junior Contributors to publish the new discussions they start without a significant increase in disruption. (see "Guardrail" below) | New discussion completion rate as defined by the number of people who click the Add topic / New section link (action = init), what % of people successfully publish at least one new discussion topic they were drafting (action = saveSuccess). What is the impact of experience level on this KPI? Note that this does not take into account the number of attempts it took for the user to publish or the duration of their editing sessions. |
| Guardrail | ...does not cause a significant increase in the percent of disruptive edits being made to talk pages | The percent of discussion topics added to talk pages that are reverted within 48 hours. The percent of editors who are blocked after adding a discussion topic to a talk page. |
| Curiosity #1 | ...causes a greater number of Junior Contributors to start participating productively on talk pages. | The number of distinct Junior Contributors who make at least one edit to a page in a talk namespace that is not reverted within 48 hours. |
| Curiosity #2 | ...causes a greater percentage of Junior Contributors continue participating productively on talk pages. | The percentage of Junior Contributors who make at least one edit to a page in a talk namespace that is not reverted within 48 hours in each of the following time intervals: 2 to 7 days after making their edit (read: within the first week), 8 to 14 days after making their first edit (read: within the second week), and 15 to 30 days after making their first edit (read: within the third or fourth weeks). |
Decision matrix
| ID | Scenario | Plan of action |
|---|---|---|
| 1. | People are more likely to publish edits using the New Discussion Tool than they are using the existing section=new interface | Continue with plans to make the New Discussion Tool available at all Wikipedias, by default. See T275256 for more detail. |
| 2. | People are less likely to publish edits using the New Discussion Tool than they are using the existing section=new workflow. | Investigate where within the New Discussion Tool workflow people are dropping off and hypothesize what could be contributing to this drop-off. In parallel, we will pause plans to make the New Discussion Tool available at all Wikipedias, by default. See: T263054#7363191. |
| 3. | People are as likely to publish edits using the New Discussion Tool as they are using the existing section=new workflow. | Continue with plans to offer the New Discussion Tool as an opt-out preference at all Wikipedias considering qualitative feedback suggests the tool is leading people to find participating on talk pages easier / more efficient. |
Done
- Finalize ===Hypotheses and ===Decision matrix
- Complete analysis
- Publish findings on mediawiki.org: Talk pages project/New discussion
Notes
- When analyzing data from the A/B test, we want to remember:
- That fa.wiki and pt.wiki have stopped allowing IPs from editing. More context in: T291306#7553319.
- To note if/when during the desktop New Discussion Tool A/B test the New Discussion Tool became available by default on mobile at the first group of wikis (T282638)
- Per the observation @MNeisler made in Slack, the way that our A/B test infrastructure is currently set up, it is possible for the following to happen:
- 1. A logged in user ("Person 1") is assigned a test bucket (e.g. test or control)
- 2. "Person 1" logs out
- 3. "Person 1" revisits the Wikipedia where the A/B test is being run
- 4. "Person 1" becomes bucketed in the A/B test for a second time which means the A/B test "thinks" that in "Step 3." someone new has entered the test when in reality it is the same person being bucketed twice
