Some additional notes:

• For any data related to false positive reports, unless there is a consistent format or some kind of template used, it will hard to scale this across wikis in an automated way.
• For "Potential revert rate average based on revert scores at various thresholds", we can show mean and mode values based on preceeding six months of data.

I think this list looks great! :)

Potential revert rate average based on revert scores at various thresholds

Let's not worry about this yet, because we might display this somewhere else (T365134).

Count of edits Automoderator failed to revert

We've decided to park this idea for now since we don't think it's going to happen very often right now.

Count of false positives reports filed

I like the idea of including this, but if it's hard to calculate we can skip it for now. We'd need to have a source of truth for where the false positive pages are besides each wiki's Automoderator config, which we don't currently have. We could then use discussion tools to count the number of threads on each page.

Proportion of false positive reports pending to be reviewed
Average duration for a false positive report to be reviewed

These are even harder because it will be hard to know for sure when a FP has been reviewed.

Added number of edits reverted as one rollback might revert more than one edit per T372299#10085680

Update:

• We have decided to split the set into monitoring metrics and key metrics.
• The key metrics will include all the above, and will be updated monthly, and the source will be wmf.mediaiki_history
• The monitoring metrics will be a subset of the key metrics, will be updated on daily basis based on MariaDB-replicas.

The reason for the split is, if Automoderator uses rollback, it will be hard to identify the edit(s) being reverted by simply matching the parent revision id. We will have to use content hashes (rev_sha1) to identify the edits being reverted. This will be very complicated to implement. Given that mediawiki_history already calculates this using revision identity-revert notion, it is best to use that. However, the issue with using mediawiki_history is that it is only updated monthly, which will be too long to wait for monitor Automoderator's activity on various wikis. So the idea is to have some metrics calculated daily, that the team and the community can use to monitor Automoderator's activity, and the key metrics can be used for reporting, evaluation etc.

Some of the monitoring metrics will be prone to some errors, for example, number of edits Automoderator has reverted. So the monitoring metrics should be used only for monitoring purposes, and the key metrics for actual evaluation. Here are proposed monitoring metrics:

• Total number of reverts made by AM
• Total number of edits reverted by AM
• Daily reverts made by AM
• Proportion of users (by user type) whose edit was reverted by AM
• Proportion of users (by user edit bucket) whose edit was reverted by AM
• Average time for Automoderator to revert an edit
• Percentage of AM's reverts reverted back
  • Potential false positive rate (excluding reverts by users' whose edit was initially reverted).

Note: The above metrics will also be updated monthly based mediawiki_history, along with daily for monitoring purposes.

On the dashboard, there'll be a separate tab for monitoring.