Page MenuHomePhabricator

Retaining Newcomers From Wikimania 2017 (Montreal) Hackathon
Closed, ResolvedPublic

Description

This task is for the technical collaboration team to discuss what we are going to do to help support and continue to engage newcomers after the Wikimania Hackathon event. Unlike Vienna, we will not target 10% because we are unsure how many newcomers there will be and we also know that Wikimania Hackathon participants are less likely to be at the event for the event's sake and may just be dropping in. Also the shorter event and the fact that many of the participants are not technical also make a different in what technical collaboration can do in the long run.
Lets talk!

Event Timeline

Rfarrand created this task.Jul 12 2017, 9:59 PM

From measuring perspective, as per discussion in T163440, we know there is no automated way to do so.

In addition to encouraging newcomers to document projects on-wiki (which most of the times leaves us with little documentation), a few changes that we could consider:

  • Format of the spreadsheet @Rfarrand shared with us for Vienna hackathon -- we add Github/ Phabricator links to separate columns, we also add columns to gather information about projects newcomers worked on, who they worked with, link to their GitHub/Gerrit commits (if there are any) from during the event, etc.
  • We also help newcomers document the same information on-wiki wherever applicable.
  • We ensure that we were able to gather information about all newcomers in the same format

We pre-assign 2-3 people for this in the same way we are assigning volunteers for other tasks.

Qgil triaged this task as Low priority.Jul 13 2017, 8:14 AM
Qgil moved this task from Backlog to Ready to Go on the Developer-Advocacy (Jul-Sep 2017) board.

(I am setting priority to "Low" only to remove "To triage"; feel free to change it.)

Very good idea. I agree that setting a specific retention goal (10% or whatever) is not the point. What we don't have and it seems that we need is a systematic way to measure onboarding and retention of new developers in relation to events.

We are already working on KPIs related to new developers and retention on a quarterly basis. There is no need to reinvent that wheel. The specific points related to events are:

  • From the data of new & retained developers captured in our KPIs, how many new and retained developers came from event X?
  • Were there other newcomers at event X that our KPIs are not registering? For instance, genuine new developers who didn't work on projects that went through code review (i.e. a new prototype hosted in GitHub).

I was thinking of a page (could be https://meta.wikimedia.org/wiki/Technical_Collaboration/Metrics) where we have a table with this information (more or less) about events organized in the past 15 months:

Name of eventDate of eventLocationNumber of new developers% active after 1 quarter2 quarters3 quarters1 yearTrend (that graph)
Wikimedia Hackathon 201719-21 May 2017Vienna (Austria)3832%17%15%11%

Events older than 15 months would be archived somewhere else. This could give us an idea about how well each event is doing, and maybe we could start seeing trends about events or locations more successful or more tought at getting and retaining newcomers.

I can already hear @Aklapper saying that we don't have tools to measure this properly. Totally true, as of today this is basically manual work. However, note that this manual work could be shared with the organizers of the events (sometimes it is Developer-Advocacy, many times it is not). This way we would also involve these organizers in the measurement of retention, and also the activities to improve retention. We can assume that most newcomers of a regular event will come from the same region where the organizers operate (i.e. Vienna & Wikimedia Austria), and therefore they also have a vested interest in good retention awareness and rates.

I can already hear @Aklapper saying that we don't have tools to measure this properly. Totally true, as of today this is basically manual work.

When registering, forcingencouraging more people to provide their email address and/or mediawiki.org user name and/or Phabricator user name would already help a lot. :)

Details on wikimedia.biterg.io only: Ever remaining problem is that even if a user had created a Phab account, as long as that account has never been active (in wikimedia.biterg.io for Phab accounts that currently means "user has created a task", not "has commented on a task" or such) it cannot be searched for in the DB hence we don't know if the person has ever registered in Phab or has registered and has had activity not indexed in the DB.

Name of eventDate of eventLocationNumber of new developers% active after 1 quarter2 quarters3 quarters1 yearTrend (that graph)
Wikimedia Hackathon 201719-21 May 2017Vienna (Austria)3832%17%15%11%

To clarify: I assume that "New developers" are not "any first-time attendees", but folks who actually proposed a code change.

  • From the data of new & retained developers captured in our KPIs, how many new and retained developers came from event X?

https://wikimedia.biterg.io/app/kibana#/dashboard/C_Gerrit_Demo lists the names of new authors in Gerrit (hence code activity only). We can manually compare these names to event registration data. However Gerrit username can again be anything, very different from mediawiki.org or Phab username...
(And maybe https://wikimedia.biterg.io/app/kibana#/dashboard/Git-Demographics would allow to get either slightly better data or an easier way, but currently cannot judge what these widgets will allow me to do until T171240 is fixed.)

  • Were there other newcomers at event X that our KPIs are not registering? For instance, genuine new developers who didn't work on projects that went through code review (i.e. a new prototype hosted in GitHub).

If we have GitHub usernames we could manually check. Looking at my GitHub stats for 2015 I can see repository names listed at least, to differentiate (as I can be active in many projects unrelated to Wikimedia).

And yes, all that sounds like a lot of manual work. :)

That is a good point, currently I am unsure if everyone on our list actually proposed a code change or not. They just self identified as newcomer and attended and were not taken off the list after review.

Qgil added a comment.Aug 7 2017, 10:49 AM

So yes, there is a lot of manual work. But there is no way around this. We are supposed to follow up on newcomers, send them feedback surveys, ask them what they need... No matter what, onboarding new developers is manual work at least in our current stage.

A required first step is to go through the list of newcomers of Vienna and Montreal, see who uploaded patches, and continue from there.

Developer Relations will be following a similar process to the process followed for Vienna.
More details coming later!

For the future - we need a better strategy to understand who we should follow up with, what we should follow up with them on and how we can help them.

Ideas - better questions in registration and a very simple survey asking about what we can do to help them / where they are stalled / what to interest them in.

Rfarrand removed Rfarrand as the assignee of this task.May 24 2018, 11:05 PM

I think this task might belong to @Aklapper or @srishakatux ?

Aklapper renamed this task from Retaining Newcomers From Wikimania Hackathon to Retaining Newcomers From Wikimania 2017 (Montreal) Hackathon .May 28 2018, 4:06 PM

I'm not sure what's left in this task that is actionable. @srishakatux: Any better clue?

(For the records, Srishti sent a survey to new developers who submitted code to Wikimedia Gerrit (which would include those Wikimania Montreal Hackathon newcomer attendees that managed to get code into Wikimedia Gerrit, and https://www.mediawiki.org/wiki/New_Developers/Quarterly/2018-01#Outreach_events and https://www.mediawiki.org/wiki/New_Developers/Quarterly/2018-04#Outreach_events list retention statistics.)

srishakatux closed this task as Resolved.EditedMay 30 2018, 11:36 PM
srishakatux claimed this task.

Yes, that's true! There isn't anything more actionable on this task. Taking the liberty to close it!