This task holds the work involved with evaluating the impacts of Paste Check through a controlled experiment with a specified start and end date.
The Editing Team is pursuing this experiment with the goal of evaluating the extent to which the feature, in its current form, warrants being deployed to all wikis.
Overarching hypothesis
If we prompt new(er) volunteers pasting text from an external site to confirm whether they wrote the content they are attempting to add, then we will see a decrease in the percentage of new content edits new(er) volunteers publish that are reverted on the grounds of WP:COPYVIO (and related policies).
Experiment timeline
WARNING: the dates and steps that follow are NOT yet final.
| Milestone | Target Completion Date | Responsible | Status | Notes |
|---|---|---|---|---|
| Publish announcement (T403637) | ✅ | @Trizek-WMF | ||
| Complete pre-deployment QA (T404923) | ✅ | @Ryasmeen | ||
| Start test (T405422) | ✅ | Editing Engineering | while the test technically started on 8 Oct, there were a few fixes that didn't land until 9 October | |
| Publish leading indicators report | ✅ | @MNeisler | ~2 weeks after test start | |
| End test | 21 November 2025 | Editing Engineering | ~6 weeks after test start | |
| Publish final report | ✅ | @MNeisler | ~2 weeks after test end | |
Decision to be made
The experiment(s) we run as part of the Paste Check are meant to help us make the following decisions...
What – if any – adjustments to the Paste Check UX need to be made (e.g. T407543) before we can be confident all of the following are true?
- Newcomers that encounter Paste Check are more likely to publish new content edits in the main namespace that are not reverted due to copyright violations (and related policies).
- Patrollers generally agree that Paste Check encourages newcomers to publish edits they consider to be constructive
KPIs
The main outcomes we are trying to impact through this feature. These are what we are primarily using for evaluating the hypothesis and deciding whether to deploy an intervention more widely.
| Hypothesis | Decision(s) to be made | Metric description |
|---|---|---|
| If we prompt new(er) volunteers pasting text from an external site to confirm whether they wrote the content they are attempting to add, then we will see a ≥4% decrease in the percentage of new content edits new(er) volunteers publish that are reverted on the grounds of WP:COPYVIO (and related policies). | Decision A: Does showing people a prompt when pasting text from an external site lower the likelihood that new content edits include copyright violations? Decision B: Do people intuitively interact with the Paste Check experience in ways that are NOT disruptive to them or the wikis? | ⭐ 1) Proportion of new content edits shown or eligible to be shown Paste Check that are reverted on the grounds of WP:COPYVIO (and related policies). Note: We decided to scope the KPI down to only new content edits that have the potential for Paste Check to show. This will allow us to focus the analysis on the edits we are seeking to impact and reliably detect any statistically significant changes within the planned AB test duration. This is based on a recent analysis of the frequency of Paste Check eligible edits and their revert rate. See T403861#11255677 2) Proportion of edits started (defined as reaching point that Paste Check was or would be shown) that are successfully published (not reverted). |
Secondary metrics
Used to learn about additional impact of Paste Check, but are not primary targets of the intervention. They reveal side effects (both positive and negative) of trying to improve the Primary Metric with the intervention.
| ID | Hypothesis | Metric description |
|---|---|---|
| Curiosity #1 | A larger proportion of new content edits by Newcomers and Junior Contributors will be constructive because they will be shown a prompt to confirm whether they wrote the content they are attempting to add when pasting text from an external site. | ⭐Proportion of published edits[i] by users with ≤100 cumulative edits that are constructive [ii] |
| Curiosity #2 | Newcomers and Junior Contributors will be more aware of the need to consider whether the text they're pasting from an external site into a main article namespace is at risk of copyright violations. | The proportion of newcomers and Junior Contributors that publish at least one new content edit that was reverted due to copyright violations Note: We’ll want to observe a decrease in this metric. |
| Curiosity #3 | Newcomers and Junior Contributors will be more likely to return to publish a new content edit in the future that does not include copyright violations because Paste Check will have caused them to realize when they are at risk of this not being true. | 1) Proportion of newcomers and Junior Contributors that publish an edit Paste Check was activated within and successfully return to make an unreverted edit to a main namespace during the identified retention period. 2) Proportion of newcomers and Junior Contributors that publish an edit Paste Check was activated within and return to make a new content edit where Paste Check was not shown during the identified retention period. |
i: We'll need to break edits out by platform as WE 1.1 is scoped to mobile-only.
ii: "Constructive edits" = edits to pages in any Wikipedia main namespace that are not reverted within 48 hours of being published
Leading indicators
T400098: [Paste Check] Report on leading indicators
Guardrails
Used to make sure that the new checks presented are not negatively impacting an editor’s experience completing an edit or causing disruption on the wikis. The scenarios named in the chart below emerged through T325851.
| Guardrail Name | Metric description | Notes |
|---|---|---|
| Edit quality decrease | Proportion of published edits that add new content and are reverted within 48 hours.Will include a breakdown of the revert rate of published new content edits shown and not shown Paste Check. | |
| Edit completion rate drastically decreases | Proportion of edits started (defined as reaching point that Paste Check was or would be shown) that are published.Will include breakdown by the number of checks shown to identify if a lower completion rate corresponds with a higher number of checks shown. | |
| People shown Paste Check are blocked at higher rates | Proportion of contributors blocked after publishing an edit where Paste Check was shown, compared to contributors not shown Paste Check. | |
| High false positive rate | Proportion of published edits where a user declined a Paste Check prompt by indicating that it was irrelevant. | Consider decision we made in T406164#11247475 to show Paste Check card on mobile immediately pasting |
Notable events
- 10 October 2025: volunteers at de.wiki revised the copy/message that appears within Paste Check
A/B Test: Decision Matrix
| ID | Scenario | Indicator(s) | Plan of Action |
|---|---|---|---|
| 1 | Paste Check is disrupting, discouraging, or otherwise getting in the way of volunteers. Read: people are less likely to publish the edits they start. | ≥20% drop in edit completion rate in edit sessions where Paste Check is activated relative to edits that would have been shown Paste Check but were not. | Pause scaling plans; If results indicate that significant decreases are only associated with a high number of Paste Checks shown, set a threshold for the maximum number checks that can be shown within a single session. If we observe significant decreases for both single and multiple checks presented in a single session, investigate changes to the UX. |
| 2 | Paste Check is increasing the likelihood that people will publish destructive edits. | Increase in the proportion of published edits where Paste Check was activated that are reverted within 48 hours relative to edits that would have been shown Paste Check but were not. Increase in the proportion of contributors blocked after publishing an edit where Paste Check was shown, compared to contributors who were not shown Paste Check. | Pause scaling plans, Review edits to try to identify any patterns in abuse and propose changes to UX to mitigate them. |
| 3 | Paste Check is causing people to publish edits that align with project policies and that are not reverted. | Decrease in the proportion of edits Paste Check was activated within that are reverted within 48 hours on the grounds of WP:COPYVIO relative to edits that would have been shown Paste Check but were not. | Move forward with scaling plans |
| 4 | Paste Check is effective at causing people to publish new content edits without pasted text from external sites, but those edits are still reverted. | Increase in the proportion of edits where Paste Check was activated that were published without unmodified pasted text AND increase or no change in the proportion of these edits that are reverted within 48 hours on the grounds of WP:COPYVIO relative to edits that would have been shown Paste Check but were not. | Pause scaling plans; Further investigation into methodology used to identify pasted text from an external site (e.g. might the false negative rate be too high); Analysis and manual review of reverted edits to understand why those edits were still reverted. |
| 5 | Paste Check is not effective at causing people to publish new content edits without pasted text from external sites but the check is not disrupting to volunteers. | No change or decrease in the proportion of new content edits Paste Check was activated within that were published without unmodified pasted text from a non-Wikipedia HTML source AND A) no significant drop in edit completion rate or B) no significant spike in block or revert rates. | Move forward with scaling plans |