Research question: What is the efficiency of Automoderator in countering vandalism on wikis?

Evaluation method: While the thresholds for success can vary based on the community, the team would consider the following as successes:

• Automoderator reverts X% of all actual vandalism [recall: TP/(TP+FN)]
• Automoderator has a baseline accuracy of 90% when reverting vandalism [precision: TP/(TP+FP)]

Baseline
To find a baseline to compare Automoderator to, we want to see if we can quantify the 'accuracy' of human patrollers. This is going to be hard to get a clear number for, but one tangible figure we want to evaluate here is how many times patrollers are themselves reverted when reverting vandalism. This would imply that they reverted an edit, but another patroller disagreed.

Question: Of all vandalism reverts, how many reverts are reverted by a different patroller?

This analysis should be done individually for the top 10 Wikipedia projects by editing activity: en, es, ja, de, fr, ru, zh, it, pt, fa, plus our pilot id.wiki

Here we will try out our operational definitions:

Vandalism

• Reverted within 12 hours
• User edit count less 15 edits
• Time since user's first edit is less than 48 hours
• Reverted by a different editor
• Edit is in the main namespace

Patroller
Users with ...
any of the following user rights:

• rollback
• review
• patrol
• block
• delete