@srishakatux: This looks like it supersedes T137214: A process to use available information about new and leaving technical contributors, or could be merged (and copying the parent task to it)?
(Also, I don't see myself realistically working on T137214 in this month so feel free to reassign if you have capacity.)

To start from the top... the name of the report. What about something self-explanatory and flexible:

New Developers Quarterly Report

Commenting on section 3.2.1 and the quantitative part:

Through Bitergia/Gerrit etc.

Important general remarks that apply to all my comments below about data from https://wikimedia.biterg.io:

• You'd need to set the time filter in the upper right corner to the last quarter/year etc.
• By default any (also paid) devs are included. To exclude, set the Advanced Filter to author_org_name:"Independent" OR author_org_name:"Unknown".
• By default Git data also includes activity from upstream repos outside of WM due to importing. To exclude, set the Advanced filter to T146135#3176718.
• As this is a quarterly activity: You can bookmark any queries on the site (so you'd only need to adjust the Time Filter 3months later) and create short URLs.

• To see a big picture: How many developers we attracted and retained every year in the last five years?

We have data for "attracted" (also see https://wikimedia.biterg.io/app/kibana#/dashboard/Git-Demographics ) but "retained" requires a definition - see also T160430#3395405 which is the same on a quarterly level.

• How many commits we received and merged in the last quarter?

If "received" means proposed commitsets in Gerrit, https://wikimedia.biterg.io/app/kibana#/dashboard/Gerrit provides a "Reviews" number.
If "merged" means changes that landed in Git, https://wikimedia.biterg.io/app/kibana#/dashboard/Git provides a "Commits" number.

• How many developers contributed in the last quarter?

If "contributed" means "in Gerrit" and hence "provided patches" (which did not necessarily get merged), https://wikimedia.biterg.io/app/kibana#/dashboard/Gerrit provides the "Submitters" number.
If this is about merged patches, see https://wikimedia.biterg.io/app/kibana#/dashboard/Git for the "Authors" number.

• How many developers actively contributed (~5 commits?) in the last quarter?

I'm not aware of an available criterion to use in the Advanced Filter and I personally doubt the usefulness, but if you really wanted, sort the "Submitters" list widget on https://wikimedia.biterg.io/app/kibana?#/dashboard/Gerrit and export it to CSV ("Export: Raw" at the bottom of the widget) and remove all contributors with less than 5 and then count the numbers (e.g. via "wc -l" on a Linux shell).

• How many new developers we attracted and retained in the last quarter?

"Attracted": See "New Authors" on https://wikimedia.biterg.io/app/kibana#/dashboard/C_Gerrit_Demo
"Retained" is KPI3 in T160430 - see https://meta.wikimedia.org/wiki/Technical_Collaboration/Metrics#Onboarding_New_Developers for data (click "Edit source" for more info as comments in the source).

• How many new developers did we onboard and retain from the developer outreach programs and events in the last quarter?

I cannot query that via Gerrit or wikimedia.biterg.io.

• Which project receives most contributions from developers?

The "Repositories" widget on https://wikimedia.biterg.io/app/kibana#/dashboard/Gerrit is sorted by "Changesets" per default.

• How many developers landed on our how-to-contribute pages in the last quarter? What were the referral paths? (Page Views)

https://tools.wmflabs.org/pageviews/?project=mediawiki.org&platform=all-access&agent=user&start=2016-03-31&end=2017-06-30&pages=How_to_become_a_MediaWiki_hacker|Starter_kit|Developer_hub|How_to_contribute offers pageview stats. Referral paths I don't know.

In general I'd love to see less indicators. For example for "Which project receives most contributions from developers?", if we gathered such info, how is it relevant and what would we use it for?

Some additional questions:

Would you recommend this project to other developers? Why or why not?
Do you feel like your contributions are valued?
What is your preferred method of communication when working on an open source project?

thank you very much @Aklapper for providing such helpful input! Looks like we are pretty much sorted with the quantitative stuff!

• How many new developers did we onboard and retain from the developer outreach programs and events in the last quarter?

I cannot query that via Gerrit or wikimedia.biterg.io.

Note to self: consider @Rfarrand's ideas in T163440#3426126 to measure this.

In general I'd love to see less indicators. For example for "Which project receives most contributions from developers?", if we gathered such info, how is it relevant and what would we use it for?

Agree! There is a hope that by learning which project receives most contributions, and from what we already know about it, we might be able to infer something. If there is anything in particular or special about this project, then perhaps we have something to learn from there.

While Ed from CE is reviewing this task, the only thing we need to agree on this task before @Qgil is on vacation -- what timeframe to consider to measure both quantitative & qualitative stuff?

Sharing my thoughts:

• For the survey:
  • I'm considering to reach out to the participants in the first week of August. And, wait until the end of the month or so before starting to consolidate the results.
  • Ideally, we keep our target audience for both quantitative and qualitative part of the survey same.
• So, for editorial purposes, May-July is the timeframe that I would like to consider.

Any concerns?

Two concerns:

• Our metrics are based on quarters as in January-March, April-June... I would think very carefully about the implication of picking different time frames for this report.
• Wikimedia Foundation's Quarterly Check-ins are usually around the 15th of the first month of each quarter (July, October....). Highlighting in those QCI data that is 2,5 months old or more seems... suboptimal?

What about this:

• The time frame or these reports are natural quarters, for instance July-September.
• For the QCI meetings we integrate the highlights only: KPIs, headlines about main news and findings. The report is being drafted publicly in a wiki page linked from the QCI documentation.
• The full report is completed and announced about four weeks after the end of the corresponding quarter (i.e. October 1st).

This way we cover quarters following the expected time frame, we present fresh information in QCI meetings, and we have one month to produce the final report.

Trying to summarize what we discussed in a video meeting:

• Calendar quarters.
• No need to get super stressed to meet deadlines (exception: KPIs at Quarterly Check-in).
• Qualitative survey could be sent at any time, does not have to cover the exact same people as the quantitative data time frame (as Andre does not expect a completely different newcomer experience and hence completely different qualitative newcomer feedback just because they started contributing in a different quarter).

Thanks to @egalvezwmf for giving such helpful feedback on the survey, sharing notes from our meeting below:

Questions review
We went through the questions with the goal to rephrase some of them or convert into quantitative questions:

3.2.2 Motivations

• How did you first hear about Wikimedia?

How did you first hear about Wikimedia as a place to contributing code to software projects? (qual.)

• What motivated you to contribute to Wikimedia? (qual.)

• How did you pick your first project to contribute to? (qual.)

Break it down:

• What was your first project? (qual.)
• Why did you decide to work on the project? (qual.)

3.2.3 Challenges

• What is that one thing that you struggled with most while making your first contribution to Wikimedia?
• Where are you currently in your understanding of the Wikimedia's code contribution process?
• What are the challenges that you encounter a lot?
• Where do you currently seek most help from when stuck in a problem while working on your project?
• What’s been most helpful to you in your journey with Wikimedia so far?
• What learning resources do you refer to for Wikimedia projects?
• How do you rate your experience contributing to Wikimedia?
• How often do you believe your contributions are utilized?

• What are the challenges that you encounter a lot? (qual)
  • What was the one thing, if anything, that you struggled with most while making your first software contribution to Wikimedia? (qual)
• Where are you currently in your understanding of the Wikimedia's code contribution process? To what extent do you agree or disagree with the following: (quant.)
  • "I have a clear understanding of Wikimedia's code contribution process"
  • Strongly agree/disagree, no opinion.
• Where do you currently seek most help from when stuck in a problem while working on your project? (qual.)
• What learning resources (training, documentation, etc) do you refer to for Wikimedia projects? (qual.)
• How likely are you to recommend Wikimedia for software contribution to friends and colleagues? (scale from 0 to 10) (quant.)
  • How do you rate your experience contributing to Wikimedia? (quant.)
• To what extent are you aware that your code contributions have been utilized? To what extent do you agree or disagree with the following: (quant.)
  • I am aware if my contributions were used or not
  • I feel my code contributions are valued
  • I feel that my opinions are considered
  • etc. etc.

See CE Insights question: 107, 108,109,110, 112, 114 - Community Engagement Team's page.

3.2.4 Needs

• What could we improve in our contribution process to support you better? (qual.)

Additional thoughts

• Concern: if we are reaching out to ~ 50 people for this survey, 25% might respond. That would be a small number, and the analysis won’t be meaningful. It would be most useful if we aggregate responses from all quarters
• Prioritize most important questions that you want to ask as in slide 12 in this presentation
• When you are not familiar with the audience, recommended way is to conduct 1:1 interviews. If it’s too much of a burden atleast 2-3 per quarter. Include an option in the survey for respondent to submit their email address if they would be interested in a follow-up conversation. Do interviews anonymously.
• Interview questions could be addressed a bit differently than survey questions, and in them, you could go deeper into the details

Is it just me thinking that these are too many questions? A too deep survey to be received by newcomers. I fear many / most will not answer it. Wouldn't it be better to have something a lot more casual but a lot more inviting, answered by the majority of newcomers?

What type of contribution are you involved in with Wikimedia? Select all that apply.

This question needs to have an "other" or a "fill in the blank" section

How often do you contribute code to Wikimedia projects?

This question needs to have more rare options and well as "other" or " fill in the blank"

To what extent are you aware that your code contributions have been utilized? To what extent do you agree or disagree with the following

Add a "not applicable" or "not relevant to me" option for each

Maybe add a question asking for ideas about where and how we should do outreach?

Might be worth getting @MelodyKramer 's opinions also.
She had some good ideas for our survey from Vienna.

[Meh. Cannot copy and paste text from qualtrics.com?]

• Are some of the questions optional / mandatory? I supposed to fill out every single thing, or will I get some error if I don't? Ah, yeah I do get "Please answer this question." and such when trying to submit. And the very last item, only after scrolling down of course, does say "(optional)".
• "Your age": I'd have expected buckets in a dropdown (like 25-34 years and such) instead of a text field but probably not important (depends on analysis)
• "Current Affiliation (e.g. Software Developer at Udacity)": I miss a way to not answer this. Or a text that does not imply that I should tell you who pays my debts.
• "gender identity": I miss a "I prefer not to answer" option.
• "type of contribution": Is there a relevant difference between "Feature development" + "Fixing bugs", and "Planning technical implementation" + "Task discussion"?
• "How often do you contribute code to Wikimedia projects?": Does "Once in a few months" make sense, depending on when the new contributor is being contacted? If I just started, how would I know how to answer? Is there a "Does not apply" (or such) option if I don't contribute code but e.g. documentation?
• "Where do you currently seek most help from when stuck in a problem while working on your project?" - do you ask me for a place like mailing lists, talk pages, IRC etc here? Because if not I don't see a big difference to "What are the challenges that you encounter a lot?"
• "To what extent are you aware that your code contributions have been utilized?" The term "utilized" confused me a bit at first as it sounded very abstract. "helpful", "accepted", something else?
• "How likely are you to recommend Wikimedia for software contribution to friends and colleagues?" Before the form talked about "code contribution"; is "software contribution" something else that has not been defined yet?

Thanks, @Rfarrand and @Aklapper :-)

Made changes accordingly! Link here.

In T167085#3539748, @Rfarrand wrote:

Maybe add a question asking for ideas about where and how we should do outreach?

There is an open-ended question on how did you first hear about Wikimedia as a place for contributing code... I am hoping we'll get some insights from this.

In T167085#3540823, @Aklapper wrote:

• Are some of the questions optional / mandatory? I supposed to fill out every single thing, or will I get some error if I don't? Ah, yeah I do get "Please answer this question." and such when trying to submit. And the very last item, only after scrolling down of course, does say "(optional)".

Only the last question is optional but for others, an error will be shown if you leave it blank and try to jump to the next section.

• "Current Affiliation (e.g. Software Developer at Udacity)": I miss a way to not answer this. Or a text that does not imply that I should tell you who pays my debts.

Do you here mean that it is not a good idea to (by the example statement XXX at YYY) ask where the new developer works? If so, how about including the following options "Student, Working Professional, Self-employed or Freelancer, Homemaker, Unemployed, etc."

• "type of contribution": Is there a relevant difference between "Feature development" + "Fixing bugs", and "Planning technical implementation" + "Task discussion"?

I think the idea here is to learn about the extent of participation. In this regard, I thought there is a difference.. in the sense that with fixing bugs, you look for something random here and there work on it, with feature development, something more concrete, continuous, long-term, and then with tech implementation planning & task discussion (on Phabricator/other venues), besides development contribution to other things too.

• "How often do you contribute code to Wikimedia projects?": Does "Once in a few months" make sense, depending on when the new contributor is being contacted? If I just started, how would I know how to answer? Is there a "Does not apply" (or such) option if I don't contribute code but e.g. documentation?

Added a "Does not apply to me because" & "Other" option with a text field

• "Where do you currently seek most help from when stuck in a problem while working on your project?" - do you ask me for a place like mailing lists, talk pages, IRC etc here? Because if not I don't see a big difference to "What are the challenges that you encounter a lot?"

Added "mailing lists, IRC, etc." as an example in the question

• "To what extent are you aware that your code contributions have been utilized?" The term "utilized" confused me a bit at first as it sounded very abstract. "helpful", "accepted", something else?

I'm using "considered" now

• "How likely are you to recommend Wikimedia for software contribution to friends and colleagues?" Before the form talked about "code contribution"; is "software contribution" something else that has not been defined yet?

Changed

In T167085#3544066, @srishakatux wrote:

Only the last question is optional but for others, an error will be shown if you leave it blank and try to jump to the next section.

Hmm. I don't think the rest of my feedback becomes less helpful if I did not want to tell you my age/gender/etc, so I'd love to generally see "Rather not tell" options...

• "Current Affiliation (e.g. Software Developer at Udacity)": I miss a way to not answer this. Or a text that does not imply that I should tell you who pays my debts.

Do you here mean that it is not a good idea to (by the example statement XXX at YYY) ask where the new developer works? If so, how about including the following options "Student, Working Professional, Self-employed or Freelancer, Homemaker, Unemployed, etc."

I don't understand yet how it's relevant who I work for. If the underlying question is whether my employer pays me for contributing to Wikimedia I'd love to have that explicit. I just don't want to be nudged into stating that I work for state authorities if state authorities might not like Wikimedia in general, for example.

• "type of contribution": Is there a relevant difference between "Feature development" + "Fixing bugs", and "Planning technical implementation" + "Task discussion"?

I think the idea here is to learn about the extent of participation. In this regard, I thought there is a difference.. in the sense that with fixing bugs, you look for something random here and there work on it, with feature development, something more concrete, continuous, long-term, and then with tech implementation planning & task discussion (on Phabricator/other venues), besides development contribution to other things too.

I see. Coming from bug wrangling it's just that sometimes differentiating between bug and feature is hard. I did not recognize the "random" between the lines. :) I could quickly drive by and post a patch to add one feature and then leave again, "scratching my own itch".

As it is now, this is a relatively extensive anonymous survey. No email address is asked, optionally. While this might work for massive surveys targeted to thousands of people, I wonder how suitable it is for our smaller and more targeted audience (new developers every quarter). Have you considered a slightly simpler survey (less questions, less options for answers, and possibility to leave email address in case we want to contact the person)?

I fear that by trying to get such precise profile with so many questions and options, we might be in fact discouraging casual participants and we are going to get fewer responses. A simpler survey might provide the same actionable data and a higher participation? If we have a way to contact participants, we could always ask them more.

All fields seem to be required. Do they need to be? This might be discouraging, and invite people to leave the survey before completing it.

I find some questions have just too many options, which also might be discouraging. I would ask ourselves, are we going to do something different with each of these different values? For instance, are we going to approach differently Masters degrees and Doctors degrees? Do we expect many developers not having finished primary school? "Other" and a blank field might keep the survey simpler, still capturing the most frequent options and leaving room for whoever wants to be precise in a different response.

Q8 What type of contributions. This is for newcomers, and therefore I wonder what "Answering developer queries" and "Planning technical implementation" really mean, and how relevant they are. Even "Task discussion" feels a bit weak when very similar options are available. On the other hand, I would ask whether they edit articles and/or contribute media to Commons, since it is interesting to know whether they are purely developers or also editors.

In other questions, answers from "very satisfied" to "very dissatisfied" (or equivalent) have 7 options. Isn't this too much? What about 5 options?

@Qgil thanks for your comments! I've made quite a few changes based on it, that has simplified the survey a bit. Removed force response in all questions, removed a question about educational qualification, added an email address question, added editing articles and upload media to commons, etc. options in a question about types of contributions, reduced number of choices in the rating questions, etc.

The TOC for the report lives here for now:
https://www.mediawiki.org/wiki/User:SSethi_(WMF)/New_Developers_Quarterly_Report_(July_-_September_2017)

In T167085#3578313, @srishakatux wrote:

The TOC for the report lives here for now:
https://www.mediawiki.org/wiki/User:SSethi_(WMF)/New_Developers_Quarterly_Report_(July_-_September_2017)

Good! I think this TOC should be a main point of discussion in our next DevRel meeting. Once we are on the same page there, it will be easier to discuss details asynchronously.

I could not join the meeting last Thursday. Meh. As requested by Srishti, I have left my feedback about the first report at https://www.mediawiki.org/wiki/User_talk:SSethi_(WMF)/New_Developers_Quarterly_Report_(July_-_September_2017)

Emails with survey link have been sent to 34 newcomers (via Qualtrics) who put their first patch into Wikimedia gerrit in July / August.
@Qgil I will create tasks that you have pointed to and 1-2 others I've in mind later today.

Note to self: Send one reminder to unfinished respondents in a week from now. Two weeks later check who started the survey but didn't finish, and send them a reminder too.

Sub: Wikimedia New Developers Survey - please participate!

Hello,

Greetings! We acknowledge your recent code contribution to a Wikimedia project! :)

We are reaching out to you as we would like to understand better what it is like to contribute to Wikimedia software projects as a newcomer, what challenges you face, and how we can help improve your experience in the future.

We would appreciate if you could take some time to fill out this survey
[link to the survey]

It will take approximately 8-10 minutes to complete. You could also choose to be interviewed anonymously via your preferred communication medium by replying to this email.

This survey is powered by a third-party service. For information about privacy and data handling, please see the survey privacy statement.

If you would like to unsubscribe from future information about the survey, follow the link to opt out of future emails below:
Click here to unsubscribe

Cheers,
Srishti Sethi
Developer Advocate
Wikimedia Foundation
https://meta.wikimedia.org/wiki/User:SSethi_(WMF)

Wikimedia Foundation
149 New Montgomery Street, 3rd floor
San Francisco, CA 94610

Reminders sent to unfinished respondents (27).