Offer the option for a ranking round. We need this option ready for testing by 2016-10-21 (if we can do it earlier, it's even better).
The backend supports creating ranking rounds and computing ranking results.
The semantics are slightly different than ratings. For instance, on the admin/round advancement side, ranking rounds can only have a max of 40 entries.
On the juror side, all ranking tasks must be gotten and completed at once. There is also no quorum concept for rankings. All active jurors must vote for the round to be finalized.
My hope is that these are easy to reflect in the UI, but if there's more I can do to clarify, please let me know.
Wait, so is it 50 or more than 50?
As long as a specific number is provided, it's not hard to change. The question is: what works well for the UI? I would argue that 50 is quite unwieldy in the UIs we've discussed (dragging to order) and juror fatigue could lead to unexpected results. If jurors don't pay attention to ordering at the bottom end, I'm not sure what inversions might occur.
My recommendation would be to have a runoff round asking "should this image be in the final round?" yes/no, with a quorum of all jurors. Should take less than an hour and can easily cut down to a healthier group.
In short, I'm happy to change the value, but I would defer to Pawel's UI design and other UX considerations.
Got you, @MahmoudHashemi. As long as the 40 can be changed, we're good. And I hear you, it's important to get the UX component right, but honestly, even 40 can be challenging for the designs we discussed earlier.
@Yarl what are our options if we don't want to have a drag-and-drop in the UI, or even with the drag and drop, how can we afford to have more photos in the ranking round?
@Effeietsanders is there an upper bound you can suggest for the ranking round? FYI, we've communicated ~50 to the countries as a good number to start the ranking round with.
Thanks for checking. I think my default approach would be:
- A pool of 50-80 images (hard upper bound 100) is available for ranking
- Each juror is asked to rank their top 10-20 (hard upper bound 25 or 30)
That leaves a bunch of images with 'no ranking' (or a shared lowest ranking).