Retain at least 10% of newcomers at the Wikimedia Hackathon 2017
Open, NormalPublic

Description

At the Wikimedia-Hackathon-2017, align current and new activities to the goal of retaining at least a 10% of new developers.

  • Offer personalized support to all newcomers at the event.

We had a very robust mentoring program at the hackathon. We advertised this program on the MediaWiki event page, by email directly to anyone who identified during registration as a newcomer, by email to all participants, and at the hackathon itself. We had multiple newcomer sessions and check ins. We had a large team of dedicated mentors at the hackathon.

  • Follow up with them after the event, connecting them to tasks and mentors.

We have followed up with newcomers by asking about their experiences in the feedback survey and are also sponsoring three of them to attend Wikimania. More direct follow up is needed in the coming months. One idea is to inform them about the new featured projects tasks and let them know that they can participate remotely in the Wikimania Hackathon.
Developer Relations will work together to determine how to check retention levels and do more follow up with the Wikimedia Hackathon newcomers.

  • Check retention levels in September and December.
Qgil created this task.Apr 20 2017, 12:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 20 2017, 12:43 PM
Qgil triaged this task as Normal priority.Apr 20 2017, 12:43 PM
Offer personalized support to all newcomers at the event.
  • We had a very robust mentoring program at the hackathon. We advertised this program on the MediaWiki event page, by email directly to anyone who identified during registration as a newcomer, by email to all participants, and at the hackathon itself. We had multiple newcomer sessions and check ins. We had a large team of dedicated mentors at the hackathon.
Follow up with them after the event, connecting them to tasks and mentors.
Check retention levels in September and December.

We have followed up with newcomers by asking about their experiences in the feedback survey and are also sponsoring three of them to attend Wikimania. More direct follow up is needed in the coming months. One idea is to inform them about the new featured projects tasks and let them know that they can participate remotely in the Wikimania Hackathon.
Developer Relations will work together to determine how to check retention levels and do more follow up with the Wikimedia Hackathon newcomers.

Qgil updated the task description. (Show Details)Jul 11 2017, 10:42 AM
Qgil added a subscriber: srishakatux.

Thank you! I think the next step is to share the list of newcomers with @srishakatux and @Aklapper (last time I checked they hadn't seen it), so they can start helping defining and executing the retention plan.

Thank you! I think the next step is to share the list of newcomers

Yes please - for example, https://wikimedia.biterg.io/ allows to check for activity by username(s) or email address.

I have re-shared the doc by email. It is the same one everyone had access to in Vienna.

I would like to start the discussion about "following up" and what is appropriate.
No newcomer opted into being emailed or followed up with even if they did opt into receiving updates about upcoming events. So because of this I do not support mass emails with a bunch of people on BCC. Yesterday when I was thinking about this I had the idea about how we can do the following up step.

If we like this plan, here are the steps that I see:

  • DR goes through the list of newcomers and really researches each person and write out everything we know about their work and interest areas.
  • Then we request to the whole team to prioritize taking 30 min each (we could even schedule a sprint time for it) and go through the list of people and make individualized recommendations based on what we know about them for next steps.
  • Then we send a personal email with both generic information as well as the personalized information to each person. (somebody from DR will have to be willing to respond to these emails if the person write back and / or connect them to a mentor.)

I don't mind sending the initial email because I think most people who attended the event would recognize my name because I sent all of the pre-event communication but in the email I would need to introduce the person to either Andre or Srishti as a contact for email questions and next steps.

Personalized emails and the rest of the plan sound great!

It would be nice if we combine this email with the one that we will send out in the next few weeks or so to gather responses for T167085. As most of these newcomers, I am assuming will meet the criteria for a target audience that we've set for the survey.

Check retention levels in September and December.

[FYI] When it comes to measuring 'activity', I'd just like to share a quick impression what https://wikimedia.biterg.io can (not) offer here:
We can track some activity (tickets created in Phab, activity on mailing lists, MediaWiki edits, patchsets, but nothing in GitHub) per person and if I know/find Phab/Git usernames or email addresses in the database behind wikimedia.biterg.io. Some examples of folks who attended the Vienna hackathon and are listed in the doc:

No good idea whether and how to actually use that data though (plus it's a bit cumbersome to look up each user handle manually in the DB).

Qgil added a comment.Jul 12 2017, 8:24 PM

Currently the list has 45 alleged newcomers, and after a review a few will probably will be removed for not being actual newcomers. I guess the first checks for activity will be done manually quarter by quarter, until we find a better process. Anyway, as the discussion goes, we are seeing that retention (at least initially, with these numbers) will be based more on personal and manual activities, rather than automated processes.

Victoria had questions about retention at our quarterly check-in, I'll try to summarise her point later, so this is more of a reminder for me to do that :)

(I have since recapped her words to DevRel and now it's up to them to reach out to her.)

@Rfarrand @Aklapper As discussed in our DR meeting last week "Srishti to proceed accordingly (contact via survey etc.) for those [not] active in Gerrit?" It's my quick guess that majority of the folks from Wikimedia hackathon will not be included in the "New Developers" study as the survey will be targeted at participants who submitted patches between July-September timeframe. So, this would apply in the context of T170495.
We can still send them a survey, but responses will not be part of our actual study. If not, then I think we don't need to wait for the survey to be ready to follow-up with the participants from Wikimedia hackathon.

Random thoughts:

One thing that we could possibly offer is an hour of 1:1 mentoring from a dev or some long term person for newcomers in their area of interest.
They can get to know someone, ask questions in a private way, maybe gain some momentum.

Even better it could be a two-part session - one where they get started and two where they get feedback and suggestions for their work.

I guess it is also important to somehow determine who actually WANTS to be involved. Just because they came to a hackathon and worked does not mean they want to spend evenings and weekends on our projects. If they don't want this at all for sure we can a) respect that b) focus more energy on others who do.

I guess it is also important to somehow determine who actually WANTS to be involved. Just because they came to a hackathon and worked does not mean they want to spend evenings and weekends on our projects. If they don't want this at all for sure we can a) respect that b) focus more energy on others who do.

Agree.. in the follow-up email, we could cover the following:

  • Appreciate their contribution at the hackathon
  • Share resources
    • New Developers page
    • Outreach programs
    • Events guide for mini events/ hackathon they want to run in their community
    • #wikimedia-devrel or link to the technical advice meeting
    • Wiki Meetups in their area
    • Participants list from the hackathon with a disclaimer
  • Ask them questions listed below:
    • Is there anybody who they met at Wikimania, and want us to connect them?
    • Is there any project that they started at the hackathon or are currently involved with, and need technical advice on it or need to be connected with a mentor?

And, then we see who gets back to us?

[FYI] When it comes to measuring 'activity', I'd just like to share a quick impression what https://wikimedia.biterg.io can (not) offer here:
We can track some activity (tickets created in Phab, activity on mailing lists, MediaWiki edits, patchsets, but nothing in GitHub) per person and if I know/find Phab/Git usernames or email addresses in the database behind wikimedia.biterg.io.

This turned out to be an "interesting problem" so I created T175854 for it.

Aklapper added a comment.EditedMon, Sep 18, 12:14 PM

Quick update on the "Get activity data of newcomers in wikimedia.biterg.io" part of this: T175854#3605693 in theory works to get the retention data I want.

  • For the Vienna Wikimedia Hackathon 2017, newcomer (based on email address, Phab username, mw.org username if provided in registration data) online activity in Wikimedia infrastructure can be seen on wikimedia.biterg.io.
  • For the Montreal Wikimania Hackathon 2017, newcomer (based on email address, Phab username if provided in registration data) online activity in Wikimedia infrastructure can be seen on wikimedia.biterg.io.

In practice, please note:

  • The displayed data is not fully reliable until T176135: Filtering on "author_name" in Git can show unrelated activity by other users is fixed.
  • Montreal data includes users' email addresses but only 5 Phab names and 0 mw.org names. If people use a different address for code etc, we cannot find their activity.
  • The two query links above include anybody who identified themselves as newcomer (also staff, also veteran volunteers, etc). To only show volunteers, click the "Independent" part of the "Organizations" diagram to filter the view. After that you could filter the view for activity of a specific person by clicking the name in the "Top Authors" widget (but note the first bullet point!).
  • In the above two query links, you cannot filter (via clicking an item) for activity of one specific newcomer only in the above two query links. In the above two query links, you can by manually editing the "Advanced Filter" field to only list a single name. After applying the "Independent" filter view (see bullet point above), you could filter the view for activity of a specific person by clicking the name in the "Top Authors" widget (but note the first bullet point!).
  • Some accounts are detached in the DB. Which is not a big issue in practice. For example, I'm going to merge the author_names "ginger", "GroundGinger" and "Ingwer, Ginger H." into a single entity once updating the DB actually works.)