Key metrics to be tracked:
Indicator | Metric(s) | Dimensions | Status |
---|---|---|---|
Volume | Number of edits being reverted by Automoderator (absolute & percentage of all reverts) | Anonymous users, newcomers[1], non-newcomers[2] | |
Accuracy (False positives) | Percentage of Automoderator's reverts reverted back | ||
Accuracy (False negatives) | Proportion of reverts not performed by Automoderator while it is turned on | - | |
Efficiency | Average time taken for Automoderator to revert an edit | - | |
- | Average time taken for Automoderator's reverts to be reverted back | - | |
Guardrail | Post deployment, proportion of edits reverted by performer | Automoderator, humans, and tool-assisted humans (if applicable) | |
The format will likely be a notebook until we eventually move to T369488: Develop a unified Automoderator Activity Dashboard (v1)