≥2 weeks after starting of the Paste Check A/B Test (T399669), we will check on a set of leading indicators (outlined below).
We will use this ticket to scope and conduct this analysis.
=== Analysis timing
Per 8 October 1:1 between Megan and Peter, we plan for the leading indicator analysis to be ready for publication by **31 October 2025**.
=== Decision(s) to be made
- [ ] What – if any – adjustments/investigations will we prioritize for us to be confident moving forward with evaluating the Tone Check's impact in T399669?
-- //E.g. might we prioritize the work necessary to unblock T407543 and enable Paste Check to be shown during the `Pre-Save` moment if people do not interact with it during `Mid-Edit`?//
=== Open questions
- [ ] 1. How might we evaluate the extent to which the options in the decline survey are misleading people? //More context in [T400098#11191795](https://phabricator.wikimedia.org/T400098#11191795).//
- [] 2. How might we reliably estimate the frequency of false negatives (instances when a user pastes text that violates copyright policies but was not shown Paste Check due to its current configuration)? We added instrumentation in T407302 to track pastes that would have caused Paste Check had the text not been copied from a known source so that we can, among other things, understand how many edits Paste Check is being suppressed from showing witin.
* //One planned analysis is to compare the revert rate of the different sources of pastes to provide insight into how many of these edit types are problematic.//
=== Leading indicators
WARNING: Need to discuss with @MNeisler
//Metrics//
|ID|Name|Metric(s) for Evaluation | Conclusion|Notes
|---|---|---|---|---|---
|1.|Newcomers and Junior Contributors are not encountering Paste Check | Proportion of new content edits Paste Check is shown within //and// Proportion of new content edits that were not eligible to be shown Paste Check because the pasted text was copied from known source (e.g. googleDocs, plainText, etc.) | | If this is low, we might consider revising the logic that prevents Check from showing on certain Pastes. //E.g. T405297.// Note: We added instrumentation in T407302 to track pastes that would have caused Paste Check had the text not been copied from a known source so that we can understand how many edits Paste Check is being suppressed from showing within.
|2.|Newcomers and Junior Contributors are not understanding the feature| Proportion of contributors that are presented Paste Check and abandon their edits| | As @Trizek-WMF noted, several de.wiki volunteers expressed concern that newcomers could be discourage by the interface copy that suggests their account could be blocked for introducing a copyright violation. This context could be helpful if we come to see a high abandonment rate.
|3.|People deem Paste Check irrelevant |Proportion of edits wherein people elect to dismiss/not change the text they've added| |Consider decision we made in T406164#11247475 to show Paste Check card on mobile immediately pasting
|4.|Paste Check is causing disruption| **1)** Proportion of people blocked after publishing an edit where Paste Check was shown //and// **2)** Proportion of published edits that add new content and are reverted within 48 hours| | In addition to edits shown Paste Check, we will also review the revert rate of edits that would have been shown Paste Check had the text not been copied from a known source. This will be logged as `ignored-paste-[source]` in VisualEditorFeatureUse.
|5.|Newcomers and Junior Contributors are not interacting with Paste Check| Proportion of edit sessions in which ≥1 Paste Check is shown and people do //not// interact [i] with at least one of the Paste Checks people were shown.| | The need for this metric emerged through T407543 wherein we identified Paste Check is //not// being shown in the `Pre-Save` moment if people do not interact with it during `Mid-Edit`
---
i. Where "interact" in this context refers to people tapping either of the buttons that appear within the Paste Check "card": {nav Yes, keep it} or {nav No, remove it}. //More in T407543#11330136.//