≥2 weeks after starting of the Reference Check A/B Test (T400101), we will check on a set of leading indicators (outlined below).
We will use this ticket to scope and conduct this analysis.
=== Analysis timing
**Analysis can begin on 18 November 2025**
NOTE: analysis will begin ≥2 weeks after starting of the Reference Check A/B Test
Decisions to be made
- [ ] 1. What – if any – **UX** adjustments/investigations will we prioritize for us to be confident moving forward with evaluating the Reference Check's impact in T400101?
- [ ] 2. What – if any – adjusts will we make to **experiment's design** to ensure enough newcomers are encountering Reference Check for us to draw statistically significant conclusions about it?
=== Leading indicators
//Metrics//
|ID|Name|Owner|Metric(s) for Evaluation | Conclusion
|---|---|---|---|---
|1.|Newcomers are not encountering Reference Check | Editing | ⭐ Proportion of new content edits Reference Check is shown within| Reference Check is shown within a sufficient number of new content edits. Reference Check was shown at least once in 42.4% of all published new-content edits by newer editors in the test group. For reference, this rate is higher than rates observed for Tone Check (9%) and Paste Check (36%).
|2.|Newcomers are not understanding the feature| Editing| ⭐ Proportion of contributors that are presented Reference Check and abandon their edits| Edits shown Reference Check are completed at a lower rate (87.1%) than eligible edits not shown Reference Check (90.6%), a 4% relative decrease. This slight decrease is not surprising as we are introducing an extra step in the workflow; as it's well below a 10% relative difference we do not see signs of concern at this time.
|3.|People deem Reference Check irrelevant| Editing |Proportion of edits wherein people elect NOT to cite the text they are attempting to add|
|4.|Reference Check is causing disruption| Editing| ⭐ **1)** Proportion of published edits that add new content and are reverted within 48hours //and// **2)** Proportion of people blocked after publishing an edit where Reference Check was shown | 1) Published new-content edits shown Reference Check are reverted less frequently, with a 13.7% relative decrease compared to eligible edits not shown the check (29.3% for the control and 25.3% for the treatment).
⭐ = Metrics we will consider required and prioritize work on first
=== Done
- [x] Summary drafted | @Iflorez
- [x] "Conclusions" documented in this ticket | @Iflorez
- [ ] Make and document paths forward for "Decisions to be made" | @ppelberg
- [ ] Findings published on mediawiki.org | @ppelberg
- [ ] Leading indicators shared with volunteers on e.wiki | @Sdkb