Page MenuHomePhabricator

Automoderator pilot metrics: Weekly reporting format
Closed, ResolvedPublic

Description

Key metrics to be tracked:

IndicatorMetric(s)DimensionsStatus
VolumeNumber of edits being reverted by Automoderator (absolute & percentage of all reverts)Anonymous users, newcomers[1], non-newcomers[2]
Accuracy (False positives)Percentage of Automoderator's reverts reverted back
Accuracy (False negatives)Proportion of reverts not performed by Automoderator while it is turned on-
EfficiencyAverage time taken for Automoderator to revert an edit-
-Average time taken for Automoderator's reverts to be reverted back-
GuardrailPost deployment, proportion of edits reverted by performerAutomoderator, humans, and tool-assisted humans (if applicable)

The format will likely be a notebook until we eventually move to T369488: Develop a unified Automoderator Activity Dashboard (v1)

Event Timeline

KCVelaga_WMF renamed this task from Automoderator pilot deployment metrics (idwiki) to Automoderator pilot deployment metrics (trwiki).Jun 10 2024, 5:17 AM
KCVelaga_WMF triaged this task as Medium priority.
KCVelaga_WMF updated the task description. (Show Details)
KCVelaga_WMF added a subscriber: Scardenasmolinar.
KCVelaga_WMF renamed this task from Automoderator pilot deployment metrics (trwiki) to Automoderator pilot metrics: Weekly reporting format.Jul 8 2024, 9:50 AM
KCVelaga_WMF updated the task description. (Show Details)

Summary of activity report until Jul 14

Reverts by Automoderator

  • From 26 June to 14 July 2024, Automoderator has made 37 reverts (at an average of 2 reverts/day, and 1 revert on most days).
  • 95% of the edits reverted by Automoderator were made by anonymous users.
  • Of the 2 edits that Automoderaotor reverted that belonged to registered users, the users belonged to 6-99 edit count bucket.
  • The average duration for Automoderator to revert an edit is 3 seconds.

Reverted Automoderator's Reverts

  • 19% (7 of 36) of Automoderator's reverts were reverted back.
    • of which 71% (5 reverts) were reverted back by same user whose edit was initially reverted.
    • of which 100% of the edits were from anonymous users.
  • On average, the time taken to revert Automoderator's revert is:
    • in cases where it was reverted by the same users whose edit was initially reverted: 148 seconds (5 reverts)
    • in cases where it was reverted by another user: 23 hours (2 reverts)
  • Potential False Positive Rate: 5%
    • not considering reverts by same anonymous user whose edit was initially reverted

Overall Revert Activity

  • On average, during the last three weeks since deployment, Automoderator handled about ~0.34% of the overall revert activity on the wiki.

Full report is available at this link


I will create a seperate task for weekly reporting updates.