In T388471, we will be inviting volunteers to review the Peacock Check language model using Annotool.

This task represents the work of updating Annotool to enable the above.

Requirements

1. Annotool is updated so that each edit/diff includes a free-text field wherein volunteers can offer context about what language they consider to be non-neutral within a given edit/diff
2. Annotool instances are created for each of the languages listed in the === Language instances section below
   • Note: the Machine Learning Team will be updating T388471 as Eval Data is ready for inclusion in Annotool.
3. Each Annotool instance will include 100 samples for languages with 3-5 evaluators, and 150 samples for languages with 6+ evaluators.

Language instances

Evaluator experience

1. An evaluator who signs up in T388471 will receive a link to the Annotool instance that pertains to the language they will be evaluating.
2. The evaluator reads the labelling instructions included in the Annotool instance.
3. The evaluator receives samples in a random order, and they are asked to review at least 30 samples. Evaluators are encouraged to review all of the samples in the dataset if they are able to, as this increases the chances that each sample will have multiple evaluators.
4. Once the evaluator has completed their labelling, they notify us in T388471.