Page MenuHomePhabricator

Design MTurk survey to test Upload Wizard instructions
Closed, DuplicatePublic

Description

Description: Ask some clickworkers to read two different sets of instructions for adding captions and descriptions to images, and see whether there is a meaningful difference in the quality of the output based on the instructions given.

Goal: Help us decide which instructions to use for redesigned Upload Wizard; evaluate whether this is a viable replacement for remote user testing for other low-level UI evaluation tasks.

Timeline: April-June 2018

Subtasks:

Subtasks:

  • select image(s) to caption
  • design MTurk task and survey
  • deploy survey with one or more sets of Turkers (different demographic groups?)
  • evaluate which instructions, on average, yielded better captions and descriptions
  • write up results

Research project page: to come

Event Timeline

Restricted Application added a project: Wikidata. · View Herald TranscriptMar 22 2018, 4:53 PM
Capt_Swing triaged this task as Medium priority.Mar 22 2018, 4:53 PM

I needed to document this in a Phab ticket for quarterly planning purposes. If a ticket for this task already exists, we can a) merge this one in or b) re-scope this one so that it just describes my specific activities related to this project.

Adding Matthias since he's working on the UW code and might be interested (mainly as FYI)

Ramsey-WMF moved this task from Untriaged to Tracking on the Multimedia board.Mar 23 2018, 6:12 PM

What's been done so far: I have selected images to caption and describe, and filled in as much of the survey as I could. Hoping @Capt_Swing can do some logic magic to make it ready for deployment.

Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Apr 23 2018, 7:19 AM