Change Details

Set up a pipeline/framework for human annotators to label relevance of different search results. The most promising approach is to use [[ https://researcher-help.prolific.com/en/collections/1059321-ai-task-builder | prolific's AI task builder tool ]]: * This allows to easily define an annotation task with a simple interface. The only requirement is a csv-file containing the data to be annotated. * It also allows to recruit participants for annotating each sample a given number of times. Different available tools from similar previous experiments; however, which we could hopefulthey cant just be used off-the-shelve and likely re-use andquire some work to adapt to ourthe current use-case * Discernatron * https://www.mediawiki.org/wiki/Discernatron * https://github.com/wikimedia/wikimedia-discovery-discernatron * Media search * https://media-search-signal-test.toolforge.org/ * https://media-search-signal-test.toolforge.org/synonyms_bak.html * https://toolsadmin.wikimedia.org/tools/id/media-search-signal-test * https://github.com/cormacparle/media-search-signal-test * Article level image suggestion * https://gitlab.wikimedia.org/toolforge-repos/alis-evaluation * https://alis-evaluation.toolforge.org/ * https://toolsadmin.wikimedia.org/tools/id/alis-evaluation * Annotool * https://annotool.toolforge.org/projects/13 * https://toolsadmin.wikimedia.org/tools/id/annotool * https://gitlab.wikimedia.org/mnz/annotool * currently only supports annotation of diffs