| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | leila | T320300 Process nominations, gather more publications and decide on the WMF-RAY 2023 winner | |||
| Resolved | KinneretG | T336615 WMF-RAY Retrospective to improve process |
Event Timeline
Comment Actions
We (Mako and Leila) held the retrospective today. Below is the summary.
What worked well
- We have a process that works.
- 2023's edition was much more efficient than the two years before that in terms of the time it took from us for reviewing, discussing, and deciding.
- We did start the process earlier than the previous years in 2023 which helped us not be rushed close to Wiki Workshop (though we missed some deadlines).
- The nomination process worked better than the years before in that we had more nominations and by more unique people.
What we want to improve for 2024
- In the call, further clarify the process for book nominations to something along the lines of "We welcome book nominations. We recommend that you specify a representative chapter of the book that the committee can start the review process from.". This is to address the point that when we have multiple book nominations, it is not reasonable to expect the committee to have read all the chapters of the book necessarily.
- Publish the COI process we follow with a particular focus on institutional COI and how we handle it. In a nutshell: If the institutional COI arises because an award committee member shares the same employer as a candidate for award, they should recuse themselves from commenting on the paper. In this case, the other committee member will make the call. (We discussed to potentially add another person to the committee to further ease the decision making, however, we did not decide to do it for 2024.)
- We discussed multiple strategies for reducing the time to review on the 200+ publications that we need to do a first pass on. The options we considered were: crowdsourcing, inviting a small number of trusted researchers who can do this first pass for us, or doing it ourselves but with more information/metadata such that we can do it more efficiently. We landed with the last solution because we expect the first one to have a large set-up cost for the first year and the second one not being appealing for people who we invite if they're not involved in the last stages of decision making. We will need help of 1-2 people who can add meta-data to the rows of the spreadsheet. The information we need is: year_of_publication, whether the work is published/pre-print/thesis/etc., list of authors, and whether it's a duplicate. We expect having this metadata allow us to speed up the stage I review process by roughly 30%. (@KinneretG and @leila to discuss options.)
- We are both excited about the idea of publishing a diff post to announce the winners and share with the Movement why this research can help them in their work.