Page MenuHomePhabricator

Work up a stat to characterize accuracy of ORES's good- and bad-faith predictions
Open, Needs TriagePublic

Description

What can we say to regular folks about the accuracy with which ORES can predict edits that are in good or bad faith? @Halfak asked me to write this task up to remind him to look into this question.

  • For good-faith edits, halfak suggested that ORES "can find 95% of good faith edits and will be right 99.8% of the time." That stat sounds good but is a little hard for most of us to grasp.
  • Meanwhile, what can we say about the accuracy of predicting bad faith edits

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 14 2016, 10:24 PM

Another question about the good-faith stat that we have at present:

ORES can find 95% of good faith edits and will be right 99.8% of the time.

The Collab Team Edit Review Improvements project is looking to identify edits that are a) likely to be damaging or reverted but which b) are in good faith. One hopes that those are not the 5% that are hard to spot, but it might be logical to suppose they would be (since, presumably, it's easier to predict that a non-problematic edit is good-faith).

What do the 5% that can't be identified look like? Is that the slice we're trying to find?

Mattflaschen-WMF renamed this task from Work up a stat to characterize accuracyof ORES's good- and bad-faith predictions to Work up a stat to characterize accuracy of ORES's good- and bad-faith predictions.Jun 20 2016, 6:53 PM