Reference Check is designed to increase the likelihood that newcomers and Junior Contributors who are editing from within Sub-Saharan Africa:
1. Publish edits that they are proud of and experienced volunteers consider useful
2. Return to edit again in the future
This task involves the work with running an A/B test (or perhaps a [multivariate test](https://en.wikipedia.org/wiki/Multivariate_statistics) [i]) to evaluate the extent to which this initial Edit Check has been effective at impacting newcomers and Junior Contributors in the ways described above.
=== Decision(s) To Be Made
- [ ] 1. Decide whether the impact Edit Check is having on users' behavior are positive enough to be made available by default, at all Wikipedias.
=== Hypotheses
|ID|Hypothesis|Metric(s) for Evaluation
|---|---|---
=== Leading indicators
//Decisions to be made//
- [ ] What – if any – adjustments/investigations will we prioritize for us to be confident moving forward with evaluating the reference Edit Check's impact?
//Metrics//
|ID|Name|Metric(s) for Evaluation
|---|---|---
|1.|Citation reuse|Percentage of people who elect to add a reference and use Citoid's {nav Re-use} feature to do so [ii]
|2.|Newcomers and Junior Conributors are not encountering Edit Check|The proportion of edits/editors the reference Edit Check is being activated within/for
|3.|Newcomers and Junior Contributors are not understanding the feature|High edit abandonment rate after reference Edit Check is shown
|4.|High volume of false positive reports|**1)** Reference Edit Check is being shown when people don't think it should be //and// **2)** High proportion of reverts among edits where reference Edit Check is shown
|5.|Reference Edit Check is causing lots of disruption|//See "3." and "4." above//
=== Guardrails
|ID|Name|Metric(s) for Evaluation
|---|---|---
=== A/B Test: Hypotheses
|ID|Hypothesis|Metric(s) for evaluation
|---|---|---
|**KPI**|The quality of new content edits newcomers and Junior Contributors make in the main namespace will increase because a greater percentage of these edits will include a reference or an explicit acknowledgement as to why these edits lack references.| **1)** Increase in the percent of published edits that add new content and include a reference or explicit acknowledgement of why a citation was not added, **2)** Decrease in the proportion of published edits that add new content (T333714) and are reverted within 48 hours (or have a low revision risk score if we use revision risk model (T317700, T343938))
|**Curiosity #1**|Newcomers and Junior Contributors will be more aware of the need to add a reference when contributing new content because the visual editor will prompt them to do so in cases where they have not done so themselves.|Increase in the percent of newcomers or Junior Contributors from SSA that publish at least one new content edit that includes a reference.
|**Curiosity #2**|Newcomers and Junior Contributors will be more likely to return to publish a new content edit in the future that includes a reference because Edit Check will have caused them to realize references are required when contributing new content to Wikipedia.|**1)** Proportion of newcomers and Junior Contributors that publish an edit Edit Check was activated within and successfully and return to make an unreverted edit to a main namespace during the identified retention period., **2)** Proportion of newcomers and Junior Contributors that publish an edit Edit Check was activated within and return to make a new content edit with a reference to a main namespace during the identified retention period.
=== A/B Test: Decision Matrix
|ID|Scenario|Plan of Action
|---|---|---
---
i. Where a "multivariate test" in this context could look like tests wherein we compare: **A)** multiple variations of Reference Check user experiences or **B)** people who are shown the source editor by default, to people who are shown VE by default, and people who are shown VE by default with Edit Check activated, as @MNeisler and @DLynch raised offline
ii. See T331582#9132480