Page MenuHomePhabricator

Document Tone Check model confidence threshold configurability on mediawiki.org
Closed, ResolvedPublic

Description

Patch: 1170418 introduces the ability for the Tone Check model confidence threshold to be configured on a per project basis.

This task involves the work of updating the Edit Check project's public documentation in ways that enable volunteers to:

  1. Know this facet of Tone Check is something that can be configured
  2. Understand how they can go about configuring this facet of Tone Check to better align with what they are seeing at the project they have been observing Tone Check on

Done

Event Timeline

I took a first pass at describing this in https://www.mediawiki.org/w/index.php?diff=7817207

Specify the confidence score required from the model for a tone check to be shown, from 0.0 - 1.0. This isn't quite "catch x% of tone violations", but rather "of the tone violations we catch, only show ones we're x% confident about". The default is 0.8, which testing suggests aligns with edit-categorization by volunteers 95% of the time.

I took a first pass at describing this in https://www.mediawiki.org/w/index.php?diff=7817207

Specify the confidence score required from the model for a tone check to be shown, from 0.0 - 1.0. This isn't quite "catch x% of tone violations", but rather "of the tone violations we catch, only show ones we're x% confident about". The default is 0.8, which testing suggests aligns with edit-categorization by volunteers 95% of the time.

Looks great. I made some edits too. Taken all together, I think we can consider this done.