Work up a stat to characterize accuracy of ORES's good- and bad-faith predictions
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	• jmatazzoni
	Jun 14 2016, 10:24 PM

Description

What can we say to regular folks about the accuracy with which ORES can predict edits that are in good or bad faith? @Halfak asked me to write this task up to remind him to look into this question.

For good-faith edits, halfak suggested that ORES "can find 95% of good faith edits and will be right 99.8% of the time." That stat sounds good but is a little hard for most of us to grasp.
Meanwhile, what can we say about the accuracy of predicting bad faith edits

Event Timeline

• jmatazzoni created this task.Jun 14 2016, 10:24 PM

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 14 2016, 10:24 PM

Another question about the good-faith stat that we have at present:

ORES can find 95% of good faith edits and will be right 99.8% of the time.

The Collab Team Edit Review Improvements project is looking to identify edits that are a) likely to be damaging or reverted but which b) are in good faith. One hopes that those are not the 5% that are hard to spot, but it might be logical to suppose they would be (since, presumably, it's easier to predict that a non-problematic edit is good-faith).

What do the 5% that can't be identified look like? Is that the slice we're trying to find?

• Mattflaschen-WMF renamed this task from Work up a stat to characterize accuracyof ORES's good- and bad-faith predictions to Work up a stat to characterize accuracy of ORES's good- and bad-faith predictions.Jun 20 2016, 6:53 PM

• jmatazzoni moved this task from Untriaged to Ready for Pickup on the Collab-Team-2016-Apr-Jun-Q4 board.Jul 5 2016, 11:43 PM

• jmatazzoni moved this task from Ready for Pickup to Untriaged on the Collab-Team-2016-Apr-Jun-Q4 board.Jul 11 2016, 9:20 PM

• jmatazzoni edited projects, added Collab-Team-Q1-July-Sep-2016; removed Collab-Team-2016-Apr-Jun-Q4.Jul 12 2016, 3:19 PM

• jmatazzoni moved this task from Untriaged to Ready for Pickup on the Collab-Team-Q1-July-Sep-2016 board.

• jmatazzoni moved this task from Backlog to Research on the Edit-Review-Improvements board.Sep 1 2016, 7:17 PM

• jmatazzoni edited projects, added Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016); removed Collab-Team-Q1-July-Sep-2016.Oct 4 2016, 10:42 PM

• jmatazzoni moved this task from Untriaged to In Development on the Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016) board.Oct 4 2016, 10:47 PM

• jmatazzoni edited projects, added Collaboration-Team-Triage; removed Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016).Jan 5 2017, 10:17 PM

• jmatazzoni moved this task from Untriaged to Old Untriaged on the Collaboration-Team-Triage board.

• jmatazzoni created subtask T164331: Define a process for adding ORES filters to new wikis when ORES is enabled on those wikis.May 3 2017, 12:09 AM

• jmatazzoni removed a subtask: T164331: Define a process for adding ORES filters to new wikis when ORES is enabled on those wikis.

• jmatazzoni moved this task from Old Untriaged to Near-Term Interest on the Collaboration-Team-Triage board.Oct 9 2017, 9:08 PM