PatruBot is too sensitive
Open, Needs TriagePublic

Description

PatruBot is reverting too many good edits. It needs to be more specific (higher precision) and less sensitive (lower recall).

Halfak created this task.Jan 31 2018, 9:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 31 2018, 9:20 PM

Generally, I think you want to set very strict criteria on a bot that automatically reverts vandalism.

E.g. with this query, you can see that setting a confidence threshold at 0.96 "probability" of damaging, you can expect to get 90% precision (wrong 1/10th of the time) and 17.7% recall (catches 17.7% of the most damaging edits):

https://ores.wikimedia.org/v3/scores/eswiki?models=damaging&model_info=statistics.thresholds.true.%27maximum%20recall%20@%20precision%20%3E=%200.9%27

Platonides added a subscriber: -jem-.