Change Details

Key metrics to be tracked: - Volume: Number of reverts (split: anonymous, newcomers, non-newcomers) - False positive: Revert rate of AM reverts by non-new editors - Pageviews of vandalized content - Proportion of edits reverted by human/AM/tool - False negatives: average risk of score of edits reverted by human patrollers A Superset dashboard can be created for most of the metrics, and rest can be reported on wiki.

Key metrics to be tracked: | **Indicator** | **Metric(s)** | **Dimensions** | **Status** | |------------------|--------------------------------------------------------------------------------|----------------------------------------------------------------------------|------------| | Volume | Number of edits being reverted by Automoderator (absolute & percentage of all reverts) | Anonymous users, newcomers[1], non-newcomers[2] | | | Accuracy (False positives) | Percentage of Automoderator's reverts reverted back | | | | Accuracy (False negatives) | Proportion of reverts not performed by Automoderator while it is turned on | - | | | Efficiency | Average time taken for Automoderator to revert an edit | - | | | - | Average time taken for Automoderator's reverts to be reverted back | - | | | Guardrail | Post deployment, proportion of edits reverted by performer | Automoderator, humans, and tool-assisted humans (if applicable) | | Note: This task is for the first iteration of tracking only. After having an initial report (for the team and the community), we can decide the cadence and form for the further monitoring.

Key metrics to be tracked: - Volume: Number of reverts (split: anonymous, newcomers, non-newcomers)| **Indicator** | **Metric(s)** | **Dimensions** | **Status** | - False positive: Revert rate of AM reverts by non-new editors|------------------|--------------------------------------------------------------------------------|----------------------------------------------------------------------------|------------| - Pageviews of vandalized content| Volume | Number of edits being reverted by Automoderator (absolute & percentage of all reverts) | Anonymous users, newcomers[1], non-newcomers[2] | | - Proportion of edits reverted by human/AM/tool| Accuracy (False positives) | Percentage of Automoderator's reverts reverted back | | | - False negatives: average risk of score| Accuracy (False negatives) | Proportion of reverts not performed by Automoderator while it is turned on | - | | | Efficiency | Average time taken for Automoderator to revert an edit | - | | | - | Average time taken for Automoderator's reverts to be reverted back | - | | | Guardrail | Post deployment, proportion of edits reverted by human patrollersperformer | Automoderator, humans, and tool-assisted humans (if applicable) | | A Superset dashboard can be created Note: This task is for the first iteration of tracking only. After having an initial report (for most ofthe team and the metricscommunity), and rest can be reported on wikiwe can decide the cadence and form for the further monitoring.