Page MenuHomePhabricator

Create New Process for Scholarship Decisions for Wikimedia Hackathon 2017
Closed, ResolvedPublic

Description

We are going to have a better defined and more clear process for deciding on hackathon scholarships in 2017. I will write up a draft plan for this eventually, trying to keep in front of the 2017 scholarship decision timeline. Please discuss ideas on this task!!!

@Chen_WMIL: any thoughts (besides the timeline) that should change from 2016?

Event Timeline

Rfarrand created this task.Apr 7 2016, 10:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 7 2016, 10:50 PM
Rfarrand triaged this task as Medium priority.Apr 7 2016, 10:53 PM
Qgil added a comment.Apr 8 2016, 6:57 AM

As I suggested in our meeting with WMAT in Jerusalem, I think the scholarship decision should be made by a committee following a documented process. I am confident that Rachel and myself have been fair in our decision process agreed every year with the local organizers, but this process is unnecessarily ad hoc, not documented, and changing slightly every year.

Elitre added a subscriber: Elitre.Apr 13 2016, 9:28 AM
Chen_WMIL added a comment.EditedApr 13 2016, 10:03 AM

The current decision making process was indeed fair and reasoned, but I agree we should have a proper documented procedure.
Putting aside the technological aspects (I'm sure you and Rachel have better knowledge and experience than I, in order to set the technological criteria), here are a few points for consideration:

  • Diversity, with emphasis on female representation
  • Returning applicants - preference should be given to people who apply again (and have not received a scholarship in the previous rounds) in light of the experience they have gained.
  • People from outside Europe and from chapters that the WMF wants to strengthen (user groups for example)
  • Participants that may contribute to projects that have been designated as high priority for the specific hackathon (as part of our discussion on selecting and initiating specific projects)
  • More balanced blend of newcomers and long-term Wikimedians
  • If necessary, and when in doubt, Skype interviews might be a good solution

Another suggestion: after the hackathon, scholarship recipients will be asked to submit a report about their work, its results etc.

Qgil added a comment.Apr 13 2016, 11:37 AM

Just for reference: https://meta.wikimedia.org/wiki/Grants:TPS

Maybe we can borrow some ideas about that process. Should applications be made on-wiki, publicly? Should reports be required? This might filter out "weak" applications from people taking just a couple of minutes to request sponsorship and try their luck. It might also deter valid candidates, though.

Qgil added a comment.Apr 13 2016, 11:41 AM

And a side comment: this task should probably apply to the Wikimedia Developer Summit as well, right? Whatever process and committee is good for the hackathon in Austria, it should be good for the Summit in USA.

Rfarrand lowered the priority of this task from Medium to Low.Apr 14 2016, 6:58 PM
Qgil added a comment.Oct 4 2016, 1:13 PM

And a side comment: this task should probably apply to the Wikimedia Developer Summit as well, right? Whatever process and committee is good for the hackathon in Austria, it should be good for the Summit in USA.

@Rfarrand, it is still not too late?

@Qgil

Not if we can create and propose a process that everyone agrees with in 2 weeks.
We need to keep to our 1 week turn around time between closing registration on Oct 23 and Oct 31.
How about involving a few representatives from the program committee as they will have the best overview of the event content and will be the best placed to make sure the person applying will find the event on-topic and useful?

As for answering your previous question about applicants requests being made public, I don't think t we can do that this time. I think it would be important to tell people in advance that their proposals will be public. If we decide this is a good idea, we can include a note in the registration for the Vienna hackathon. The problem with that is that not everyone who might request a scholarship if it were private would feel comfortable requesting if they knew that even the fact that they requested was public - much less the content.
Requiring reports could be a good idea - but a few things things would need to be in place: Somebody to enforce the deadline and also we would need to be willing to stick to some consequence (no scholarships from us for anything for two years as an example) and the the issue would be that it would need to be something useful - and not just a busy-work form. Maybe we could give them option like 1) write a blog about the event somewhere and send us a link, 2) run a tech-talk about something you worked on or learned about at the event 3) help us track and summarize 5 different projects that took place at the event.

Qgil added a comment.Oct 11 2016, 10:11 AM

OK, so for Wikimedia-Developer-Summit (2017) it is still @Rfarrand and @Qgil making decisions, seeking advice from the Program committee when needed.

OK!
So we are currently working on a process for the Developer Summit here:
https://phabricator.wikimedia.org/T141937

I believe it is a major improvement on anything we have done in the past.
Additionally we have also begun tracking who has received scholarships in the past and will begin weighting this in the Hackathon scholarship decision making process as well.

After the Dev Summit we will review the scholarship process with WMAT and see if everyone likes the plan. More details here after the Dev Summit.

Rfarrand added a subscriber: JeanFred.EditedJan 18 2017, 1:19 AM

Adding @JeanFred to this task as he im managing this process for 2017 instead of the Technical Collaboration team.
A lot of the process is being taken from the 2017 Wikimedia Developer Summit process although I suspect there will be some changes.
Whenever you mail or publish anything about our process for this year can you add a link to this task please Jean Fred? :)

The scholarship process for 2017 went well.
Here are the lessons learned copy pasted from this etherpad: https://etherpad.wikimedia.org/p/Hackathon_Scholarship_Committee

Retrospective

What went well
  • Use a single online spreadsheet. Low budget, versatile, consistent.
  • Start rating applications as early on as possible, but still make sure all applications within the deadline get enough exposure. This lowered the load over time.
  • Prioritize the scoring of the application requiring visa.
  • Use pivot tables and other spreadsheet magic to assess ratios. Gender balance (at least 20% women), attendees from Japan, Africa.
  • Consider reducing "group applications" of unexperienced people to approving a single attendee (Cameroon, Albania) to try and seed those communities.
  • Prepare well (rate applications, add travel costs, make rankings available). This allows being very productive in an approval meeting. For the Vienna hackathon, 1 meeting of 75 minutes was needed. The rest was done over a limited numbers of email threads with little noise and in spreadsheets with comments.
  • Provide a certain level of explanation to your votes. This fosters early, asynchronous conversations, which speeds up the approval meeting.
  • Consider having representatives of the scholarship funders on the scholarship committee. Ensures representation of the funder, allows last minute prioritisation when you can or must, adds understanding of meaning.
What can be done better
  • Make estimates for total sponsorship costs of a candidate early on. We had to look up flight cost estimates last minute, and there were comments during the approval meeting that short flights within Europe were estimated way too high. Both are disrupting items in the approval meeting.
  • Have clarity about room costs. There was unclarity about the actual budget costs of the hotel rooms, and we found out that hotel rooms with single occupancy were really costly. We decided to calculated with a lower rate, but this introduced a significant overall cost risk. We asked a question by email to the organizers, that was replied to quickly, but in the approval meeting itself, this issue wasn't resolved and calculated. From what we can tell now, this resulted in possibly approving too many people.
  • Make clear that shared rooms are the norm for scholarship recipients in the application form. It's still unclear if this is actually the case, but we feel like it is. During the Vienna hackathon, there is for example a €270 different in single vs. shared room cost (total €450 for shared and €780 for single). If shared is the norm, this allows more scholarships to be awarded.
  • Follow through with early approval of high scoring applications requiring visa. This was the plan, but it seems like this slipped by about a week. It would be a shame if visa approval would fail because of unnecessary time lost.
  • Have a clear planning for the complete process, especially of when early approval needs to be ready, and when this should be communicated. Setting milestones helps in getting things done and communicating urgency where needed.
  • Budget availability from all donors should be clear before the approval meeting. Additional budget availability was communicated during the approval meeting. This might have been clear earlier on in the process, because it caused some confusion.
  • Have a kick-off meeting for the scholarship committee. This meeting is used to explain the process, the timeline, and resolve any questions the members would have.
What still puzzles us
  • What ratios are important for our event?
    • Number of mentors
    • Gender diversity
    • Geo diversity
  • What is the actual mandate of the scholarship commitee?
    • Shall we rate some folks lower because we feel other orgs should fund them? IEG granted projects, Kiwix, people from countries with chapters with significant budgets.
Rfarrand closed this task as Resolved.Mar 16 2017, 9:31 PM
Qgil awarded a token.Mar 17 2017, 4:45 PM