Set up a pipeline/framework for human annotators to label relevance of different search results.

The most promising approach is to use prolific's AI task builder tool:

• This allows to easily define an annotation task with a simple interface. The only requirement is a csv-file containing the data to be annotated.
• It also allows to recruit participants for annotating each sample a given number of times.

Different available tools from similar previous experiments; however, they cant just be used off-the-shelve and likely require some work to adapt to the current use-case

• Discernatron
  • https://www.mediawiki.org/wiki/Discernatron
  • https://github.com/wikimedia/wikimedia-discovery-discernatron
• Media search
  • https://media-search-signal-test.toolforge.org/
  • https://media-search-signal-test.toolforge.org/synonyms_bak.html
  • https://toolsadmin.wikimedia.org/tools/id/media-search-signal-test
  • https://github.com/cormacparle/media-search-signal-test
• Article level image suggestion
  • https://gitlab.wikimedia.org/toolforge-repos/alis-evaluation
  • https://alis-evaluation.toolforge.org/
  • https://toolsadmin.wikimedia.org/tools/id/alis-evaluation
• Annotool
  • https://annotool.toolforge.org/projects/13
  • https://toolsadmin.wikimedia.org/tools/id/annotool
  • https://gitlab.wikimedia.org/mnz/annotool
  • currently only supports annotation of diffs