Improve Key performance indicator: code contributors new / gone
Closed, ResolvedPublic

Description

http://korma.wmflabs.org/browser/code_contrib_new_gone.html mjust answer these questions:

Who are the new code contributors (commits + reviews)? Are they increasing their involvement? Who seems to be on a way out or gone? How are our contributor intake & loss evolving?

  • Number of new contributors with 1 / 2-5 / 6+ changes submitted in the past 3 months (values may be fine tuned based on actual data).
  • (How to register increasing engagement versus one-offs or new contributors disengaging and vanishing after a short period?)
  • Number of contributors stopping contributing or decreasing continuously in the past 3 months.

Version: unspecified
Severity: normal

Details

Reference
bz61563
bzimport raised the priority of this task from to Normal.
bzimport set Reference to bz61563.

By the way, I've been using this report for some months: http://www.mediawiki.org/wiki/Gerrit/Reports/Open_changesets_by_newbie_owner
I welcome the new contributors and give some pointers when necessary, add reviewers when missing. Often, if not in most cases, they receive some code review within the day (!); sometimes even a +2, but more often requests by submitter or reviewer rot unaddressed for months and the backlog grows.

Acs added a comment.Mar 4 2014, 5:55 PM

Updated this KPI. Now there are two tables, one for newcomers and a second one for people gone. Both follow the same format:

http://korma.wmflabs.org/browser/code_contrib_new_gone.html

Nemo, in this report you have some more info comparing to your current report. You can see is a person is having probs (no merges in one moth, or abadons ...) and also, yo can see the first date for contribution and a brief activity graph. In next iteration we plan to add ordering for columns, as you have now in your report.

Nemo, thoughts about the current of korma report?

(In reply to Alvaro from comment #2)

Nemo, thoughts about the current of korma report?

Not really. I sent https://github.com/Bitergia/mediawiki-dashboard/pull/41
The first table is rather straightforward, I've not really tried to understand/analyse the second table and the graph.

The tables and graphs are now considered ready:

http://korma.wmflabs.org/browser/code_contrib_new_gone.html

We are still missing better strings, but this is a task for me. I'm taking the bug. Help / patches welcome.

(In reply to Nemo from comment #3)

The first table is rather straightforward, I've not really tried to
understand/analyse the second table and the graph.

The second table show contributors that haven't submitted any patch in the last 6 months. I'm not especially happy about the results it shows, but we can fine tune them, and file separate enhancement requests. Is 6 months a too long period (when we identify them, they are long gone)? Should reviews and comments also be counted (maybe someone hasn't upload a new changeset in 6 months, but they are reviewing others'patches regularly)?

Similar thoughts with the graph: it's a good first step, but we can probably improve it. The idea is to show the intake of new contributrs vs the established community; we want to see that we are not stagnant, and hopefully having a healthy trend of newcomers. Should we count only the last 12 months to have a fresher and more dynamic picture of the new contributors versus the established ones? Should we add a line for the number of contributors with more than 10 patches? Should we use a cumulative graph, like "How is the weight of the WMF evolving?" at http://korma.wmflabs.org/browser/who_contributes_code.html ? Separate enhancement requests are welcome.

Qgil lowered the priority of this task from Normal to Low.Nov 23 2014, 11:27 PM
Qgil removed Qgil as the assignee of this task.Dec 16 2014, 8:16 AM
Qgil raised the priority of this task from Low to Normal.Jun 25 2015, 8:49 AM
Aklapper renamed this task from Key performance indicator: code contributors new / gone to Improve Key performance indicator: code contributors new / gone.Aug 27 2015, 10:40 AM
Aklapper set Security to None.
Aklapper updated the task description. (Show Details)Nov 9 2015, 11:40 AM
Aklapper lowered the priority of this task from Normal to Low.
Aklapper claimed this task.

The page does not really show any trends for a specific contributor. I'm not even sure what its scope should be.

6 months of no contribution is too much - if I have not contributed for six months I'm pretty much out.

http://korma.wmflabs.org/browser/code_contrib_new_gone.html has three columns listing a user's number of patches which have been

Submitted Merged Abandoned

I want new contributors to receive feedback but there's no data whether there has been a review for a certain patch, or if there has been no review at all (which means we need to ping on such patches and find reviewers).

Related code is in mediawiki-dashboard/browser/js/mediawiki.js and GrimoireLib/vizgrimoire/analysis/contributors_new_gone.py

Aklapper added a comment.EditedDec 21 2015, 8:21 PM
In T63563#678030, @Qgil wrote:

Number of contributors stopping contributing or decreasing continuously in the past 3 months.

I'd say that's what http://korma.wmflabs.org/browser/demographics.html is for instead.

Some thoughts on each of those three existing sections (see screenshot above):

  • "Who are the new code contributors (commits + reviews)? People with first submission in last tree months. Details for the the first submission."
    • TODO: CHECK CODE: Clarify what is a "submission". Changeset? Patchsets within one patchset? Patchsets+potential review comments?
      • In case "reviews" are included, I think I don't see why I'm interested if a new contributor adds reviews.
    • List shows number of Submitted / Merged / Abandoned" changesets (?) and date of first submission. It does not show me how many changesets of that user have received zero feedback (CR=0) to allow following up and making sure the contributor receives review within a certain timeframe.
      • TODO: Add another column for "Unreviewed" with (status:open and label:Code-Review=0 and label:Verified=+1), preferably being a direct link to the Gerrit URL for that query?
    • Harder to solve in that UI, hence let's ignore for now: For CR=0 changesets I cannot see how old each of their unreviewed changesets.
    • Typo: "tree" → "three"
  • "Who seems to be on the way out or gone?"
    • Currently not useful because we have no concept to follow up on these leaving/gone contributors (yet). Just saying.
      • TODO: How is it helpful for us to know which contributors are on the leave and how do we use that information?
  • "People Intake" graph
    • I don't consider this graph useful because I have no idea what it is supposed to tell me. TODO: Some graph goes up or down: What does it mean? Numbers are per month but what do they express? "In month 11/2014, there were 50 users who had each submitted a total of 5-9 patchsets within any timeframe (first patchset might have been in 2002)"? If so, I'm surprised the graphs don't increase. And not increasing means that many many people manage to get over 10 patches which I doubt. In any case, clarify the description.
      • TODO: CHECK CODE?
    • Description mentions "loss evolving" but that is not covered at all. And "Loss: how many people loss now, six months ago". Remove both.
    • If considered somehow useful: "People Intake: Number of people with 1 submission, 2-5 submissions, 6-10 submissions." - TODO: Clarify what is a submission. Changeset? Patchsets within one patchset? Patchsets+potential review comments?
    • Typo: "Inktake" → "Intake"
  • The "Graph of Activity" has no scales and currently does not not help me realize anything.
    • TODO: Add scale? On which other page can a similar graph be seen, by the way?

....and on http://korma.wmflabs.org/browser/code_contrib_new_gone.html I have to manually do the math to subtract "Abandoned" and "Merged" from "Submitted", and after doing so I still cannot identify patches without any review (V-2, CR+1 or CR-1) that we should find a reviewer for.

That page is lovely, and I wonder which parts/aspects to "take over" to make http://korma.wmflabs.org/browser/code_contrib_new_gone.html better.

TODO: I don't know why "Abandoned" is interesting but I'd rather go for these columns:

Total SubmittedCR=0 && V=1MergedCR-1CR+1
Aklapper lowered the priority of this task from Low to Lowest.Jan 28 2016, 9:05 PM

I've sorted out above in T63563#1895837 and T63563#1979517 what I see as TODOs.

The current page kind of works for my main interest: To be able to see patchsets by new contributors and take a look at them and to try making them get a timely review and reviewers subscribed (thanks to everybody else who's also trying to do this). If I do the math on my own and click a bit more on the current page, I can achieve that.

Given that we're after killing Gerrit and going for Differential, I will not add this task to DevRel-February-2016 (next month) and I do not consider this task a priority.
We could re-evaluate this task once a Differential backend is available in MetricsGrimoire / korma.

Aklapper removed Aklapper as the assignee of this task.Jan 28 2016, 9:05 PM

If the current page kind of works for your purposes, then maybe we can just resolve this task? I don't see the benefit to keep it open...

If the current page kind of works for your purposes, then maybe we can just resolve this task?

Probably. Maybe remove "Improve" from the summary, as that would be an open-ended task.

Aklapper closed this task as Resolved.Feb 1 2016, 8:40 PM
Aklapper claimed this task.

Yeah. Resolving.

To potentially re-evalulate after fixing T118753.