≥2 weeks after starting of the Tone Check A/B Test (T387918), we will check on a set of leading indicators (outlined below).
We will use this ticket to scope and conduct this analysis.
=== Analysis timing
!!TBD!!
=== Decision(s) to be made
What – if any – adjustments/investigations will we prioritize for us to be confident moving forward with evaluating the Peacock Check's impact in T387918?
=== Leading indicators
WARNING: Need to discuss with @MNeisler and @SSalgaonkar-WMF
//Metrics//
|ID|Name|Owner|Metric(s) for Evaluation | Conclusion
|---|---|---|---|---
|1.|Newcomers and Junior Contributors are not encountering Peacock Check | Editing | Proportion of new content edits Peacock Check is shown within|
|2.|Newcomers and Junior Contributors are not understanding the feature| Editing| Proportion of contributors that are presented Peacock Check and abandon their edits|
|3.|People deem Peacock Check irrelevant| Editing |Proportion of edits wherein people elect to dismiss/not change the text they've added|
|4.|Peacock Check is causing disruption| Editing| **1)** Proportion of people blocked after publishing an edit where Peacock was shown //and// **2)** Proportion of published edits that add new content and are reverted within 48hours|
|5.| People are encountering Peacock Check two late in editing journey | Editing | Proportion of Peacock Checks shown Mid-Edit compared to Pre-save|
|6.| Model is not able to evaluate tone of published edit quickly enough| ML| Proportion of edits that are published before the model is able to return an evaluation. See T388716.|
|7.| Model service availability| ML| Service Availability SLO: 95% of all requests return a 200/300/400 response code. See T390706|