≥2 weeks after starting of the Paste Check A/B Test (T399669), we will check on a set of leading indicators (outlined below).
We will use this ticket to scope and conduct this analysis.
=== Analysis timing
Per 8 October 1:1 between Megan and Peter, we plan for the leading indicator analysis to be ready for publication by **31 October 2025**.
=== Decision(s) to be made
- [ ] What – if any – adjustments/investigations will we prioritize for us to be confident moving forward with evaluating the Paste Check's impact in T399669?
-- //E.g. might we prioritize the work necessary to unblock T407543 and enable Paste Check to be shown during the `Pre-Save` moment if people do not interact with it during `Mid-Edit`?//
=== Open questions
- [ ] 1. How might we evaluate the extent to which the options in the decline survey are misleading people? //More context in [T400098#11191795](https://phabricator.wikimedia.org/T400098#11191795).//
- [] 2. How might we reliably estimate the frequency of false negatives (instances when a user pastes text that violates copyright policies but was not shown Paste Check due to its current configuration)? We added instrumentation in T407302 to track pastes that would have caused Paste Check had the text not been copied from a known source so that we can, among other things, understand how many edits Paste Check is being suppressed from showing witin.
* //One planned analysis is to compare the revert rate of the different sources of pastes to provide insight into how many of these edit types are problematic.//
=== Leading indicators
WARNING: Need to discuss with @MNeisler
//Metrics//
|ID|Name|Metric(s) for Evaluation | Conclusion|Notes
|---|---|---|---|---|---
|1.|Newcomers and Junior Contributors are not encountering Paste Check | Proportion of new content edits Paste Check is shown within //and// Proportion of new content edits that were not eligible to be shown Paste Check because the pasted text was copied from known source (e.g. googleDocs, plainText, etc.) | 🟩 **No action needed at this time:** paste Check is being shown in a sufficient number of the new content edits newcomers in the test group made (36%). For reference, this rate is significantly higher than rates observed for Tone Check (9%). | If this is low, we might consider revising the logic that prevents Check from showing on certain Pastes. E.g. T405297. //Note: We added instrumentation in T407302 to track pastes that would have caused Paste Check had the text not been copied from a known source so that we can understand how many edits Paste Check is being suppressed from showing within.//
|2.|Newcomers and Junior Contributors are not understanding the feature| Proportion of contributors that are presented Paste Check and abandon their edits| 🟩 **No action needed at this time:** newcomers presented Paste Check are doing the //opposite// of abandoning edits, they are completing them at a higher rate (52%) than edits in the control group that are eligible but not shown Paste Check (49%). | As @Trizek-WMF noted, several de.wiki volunteers expressed concern that newcomers could be discourage by the interface copy that suggests their account could be blocked for introducing a copyright violation. This context could be helpful if we come to see a high abandonment rate.
|3.|People deem Paste Check irrelevant |Proportion of edits wherein people elect to dismiss/not change the text they've added| 🟩 **No action needed at this time:** people are dismissing Paste Check at a rate (55%) that is similar to rates we observed for Tone Check and Reference Check.|Consider decision we made in T406164#11247475 to show Paste Check card on mobile immediately pasting
|4.|Paste Check is causing disruption| **1)** Proportion of people blocked after publishing an edit where Paste Check was shown //and// **2)** Proportion of published edits that add new content and are reverted within 48 hours|🟩 **No action needed at this time:** overall, new content edits shown Paste Check are reverted //less frequently.// We've observed a -21.3% decrease in published edits where Paste Check was shown compared to edits eligible but not shown | In addition to edits shown Paste Check, we will also review the revert rate of edits that would have been shown Paste Check had the text not been copied from a known source. This will be logged as `ignored-paste-[source]` in VisualEditorFeatureUse.
|5.|Newcomers and Junior Contributors are not interacting with Paste Check| Proportion of edit sessions in which ≥1 Paste Check is shown and people do //not// interact [i] with one or more of the Paste Checks that were were shown.| 🟩 **No action needed at this time:** in more than half of of all editing sessions where Paste Check was shown (55%), people interact with one or more of the Paste Checks presented. This rate is similar to those we observed with Reference Check and Tone Check. We'll revisit the priority of interventions to potentially increase this rate (e.g. T407543) following the completion of the final analysis (T399669) | The need for this metric emerged through T407543 wherein we identified Paste Check is //not// being shown in the `Pre-Save` moment if people do not interact with it during `Mid-Edit`
---
i. Where "interact" in this context refers to people tapping either of the buttons that appear within the Paste Check "card": {nav Yes, keep it} or {nav No, remove it}. //More in T407543#11330136.//