**Research question:** What is the efficiency of Automoderator in countering vandalism on wikis?
**Evaluation method:** While the thresholds for success can vary based on the community, the team would consider the following as successes:
- Automoderator reverts X% of all actual vandalism [recall: TP/(TP+FN)]
- Automoderator has a baseline accuracy of 90% when reverting vandalism [precision: TP/(TP+FP)]
**Baseline**
To find a baseline to compare Automoderator to, we want to see if we can quantify the 'accuracy' of human patrollers. This is going to be hard to get a clear number for, but one tangible figure we want to evaluate here is how many times patrollers are themselves reverted when reverting vandalism. This would imply that they reverted an edit, but another patroller disagreed.
**Question:** Of all vandalism reverts, how many reverts are reverted by another patroller?
Here we will try out our operational definitions:
**Vandalism**
- Edit from account with <25 edits or anonymous user
- Reverted by a different editor
- Revert happens within 24 hours
- Edit is in the main namespace
**Patroller**
Users with ...
any of the following user rights:
- rollback
- review
- patrol
- block
- delete