Page MenuHomePhabricator

Define criteria for human editor verification of MV-generated metadata for purposes of model refinement
Closed, InvalidPublic

Description

It is likely that labeling providers will provide an API for submitting human verification data in order to refine the labeling model.

  • How many editors must approve a label before it is considered verified? Only the uploader, or more?
  • What percentage of human reviewers shown the label must agree that a given label applies?

See also T227342: Define criteria for human editor verification of MV-generated metadata for promotion to SDC

Event Timeline

Mholloway created this task.Jul 5 2019, 4:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 5 2019, 4:30 PM
Mholloway triaged this task as Medium priority.Jul 17 2019, 3:52 PM
Mholloway lowered the priority of this task from Medium to Low.Jul 18 2019, 3:20 PM

I think it's safe at this point to deprioritize third-party model feedback relative to other tasks.

Mholloway closed this task as Invalid.Aug 1 2019, 9:33 PM

This will depend on the provider/API. For now all we need to do is record votes.