Visually report damaging confidence
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Halfak
	Sep 7 2016, 2:14 PM

Description

From https://www.mediawiki.org/wiki/Topic:Tb34n8tdmpv4vc38

the red in my watchlist screams "red alert!" big problem! Lots of likilhood of it being terrible" Wheras my experience so far has been more in a warmer colour (orange, or something), where the change is of need for attention but not screaming at me.

We should set 3 thresholds for color:

filter_rate_at_recall(min_recall=0.9): yellow (review for completeness)
filter_rate_at_recall(min_recall=0.75): orange (likely to be damaging)
recall_at_fpr(max_fpr=0.1): red (almost certainly damaging)

In the case of English Wikipedia's damaging model, this would set the thresholds to (20%, 46%, 94%).

It would be great if we also had some sort of tooltip that read the exact prediction probability like the ScoredRevisions tool. E.g. "85% damaging, 23% goodfaith"

Result:

Details

	Subject	Repo	Branch	Lines +/-
	Visually report damaging confidence	mediawiki/extensions/ORES	master	+49 -4
	Expose ORES damaging thresholds in javascript	mediawiki/extensions/ORES	master	+5 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Ladsgroup	T144922 Visually report damaging confidence
Resolved	Ladsgroup	T143611 Embed machine readable ores scores as data on pages where ORES scores things
Resolved	• Catrope	T137966 Include goodfaith model information in ORES review tool

Event Timeline

Halfak created this task.Sep 7 2016, 2:14 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 7 2016, 2:14 PM

Halfak updated the task description. (Show Details)Sep 7 2016, 2:16 PM

@Pginer-WMF is exploring some design concepts for representing confidence in T138935. In this case, he's looking to include a flag widget that would express the confidence. So the simple "r" would be replaced with something that like: [Damaging ••○]

Halfak edited projects, added Machine-Learning-Team; removed Machine-Learning-Team (Active Tasks).Sep 7 2016, 2:26 PM

Thanks for making this a phabricator item! Looking forward to the update on how this works!

He7d3r subscribed.Sep 7 2016, 6:28 PM

Halfak added a subtask: T143611: Embed machine readable ores scores as data on pages where ORES scores things.Sep 8 2016, 2:38 PM

Halfak triaged this task as High priority.Sep 8 2016, 2:43 PM

Halfak moved this task from Unsorted to New development on the Machine-Learning-Team board.

Halfak moved this task from Backlog to Prioritized on the MediaWiki-extensions-ORES board.

Halfak added a subtask: T137966: Include goodfaith model information in ORES review tool.

Ladsgroup claimed this task.Sep 14 2016, 4:57 PM

Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 14 2016, 4:57 PM

Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Sep 14 2016, 4:59 PM

P4289 (An Untitled Masterwork)

2345 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

class="phui-tag-core-closed">T143611: Embed machine readable ores scores as data on pages where ORES scores things is not deployed yet but once it's deployed we can simply run fun javascripts to make recent changes more colorful. For example I wrote this:
class="paste-embed-body" style="max-height: 27.6em;">

/** class="cm"> * Created by amir on 10/23/16. class="cm"> */ class="p">( function ( mw, $ ) { 'use strict'; var colors = { 0.40: '#750787', 0.50: '#004DFF', 0.60: '#008026', 0.70: '#FFED00', 0.80: '#FF8C00', 0.90: '#E40303' } $('li').each( function (){ if ( $( this ).children('a').attr('href') ) { var reg = /diff=(\d+)/ig var res = reg.exec( $(this).children('a').attr('href') ); if (res && res[1] in mw.config.get('oresData')) { var score = mw.config.get('oresData')[res[1]]['damaging']; var threshold = 0; for ( threshold in colors ) { if ( score > threshold ) { $( this ).css( 'background-color', colors[threshold]); } } } } } ) class="p">}( mediaWiki, jQuery ) );

Which made this:

(Rainbows!)
Now we should talk about how we can use this in the extension.

Note, we talked about this in the revscoring meeting and we determined that three thresholds should be surfaced through a config variable from https://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php

The new threshold will come from recall_at_fpr(max_fpr=0.1).

Halfak closed subtask T143611: Embed machine readable ores scores as data on pages where ORES scores things as Resolved.Oct 24 2016, 7:33 PM

Pginer-WMF added a subscriber: • jmatazzoni.Oct 27 2016, 12:29 PM

In the recent iteration on the exploration we are doing to integrate ORES filters and others into Recent Changes (T147632), a flexible highlighting mechanism is provided. Users can define what to highlight and the colors to use. The colors allow for the three common colors used for warnings (red, orange and yellow) and the filters provided allow to target three filtering levels based on different precision/recall values for damaging edits. We also used the list bullet points to clarify the cases where more than one highlighting criteria applies to a given row.

On the testing sessions, the system seems to work well and users appreciate the metaphor of highlighting as they would do in a paper-base document so far. But more details will be shared as @dchen completes the study.

Currently we were not adding any highlight by default, but we can consider doing so if we consider any to be generally relevant for all users of Recent Changes. In any case, filtering settings are expected to be reflected in the URL so it is possible to direct users to the Recent Changes page with a specific set of filters and highlights form a context where those are expected.

Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Oct 30 2016, 2:06 AM

Change 318774 had a related patch set uploaded (by Ladsgroup):
Expose ORES damaging thresholds in javascript

https://gerrit.wikimedia.org/r/318774

gerritbot added a project: Patch-For-Review.Oct 30 2016, 2:07 AM

Ladsgroup moved this task from In progress to Blocked on others on the User-Ladsgroup board.Oct 30 2016, 2:07 AM

Re: the colors, it would be good to have some defaults or recommendations, for color schemes that are accessibility-friendly.
I took a quick attempt, with the filters at http://colorbrewer2.org/#type=diverging&scheme=RdYlBu&n=6 (+ checkmark "colorblind safe" and increase the transparency so that text overlaid is highly legible) and produced:
#d73027
#fc8d59
#fee090
#e0f3f8
#91bfdb
#4575b4

Pau or Volker might have better suggestions on how to find a broader selection - (that map tool only offers 6 colors, if the "colorblind friendly" setting is checked).

Ladsgroup moved this task from Blocked on others to In progress on the User-Ladsgroup board.Nov 5 2016, 12:39 PM

Ladsgroup moved this task from Review to Parked on the Machine-Learning-Team (Active Tasks) board.

Change 318774 merged by jenkins-bot:
Expose ORES damaging thresholds in javascript

https://gerrit.wikimedia.org/r/318774

ReleaseTaggerBot added a project: MW-1.29-release (WMF-deploy-2016-11-08_(1.29.0-wmf.2)).Nov 5 2016, 1:00 PM

Change 320341 had a related patch set uploaded (by Ladsgroup):
Visually report damaging confidence

https://gerrit.wikimedia.org/r/320341

Ladsgroup moved this task from In progress to Blocked on others on the User-Ladsgroup board.Nov 8 2016, 2:38 AM

Ladsgroup moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.

Volker_E added a project: Accessibility.Nov 16 2016, 9:22 AM

Volker_E subscribed.

SBisson mentioned this in T149467: Build user interface for the Highlight Tools and implement highlighting in the Edit Results List.Nov 17 2016, 5:24 PM