Most basic Tech Community metrics are published and up to date
Closed, ResolvedPublic

Description

Intention and Principles

  • Metrics (charts) show clear trends over time (instead of an absolute snapshot number without context to compare with). It should be immediately clear and shared understanding if it's good or bad when a chart goes up or down.
  • People are more important than objects. Activity related to code (engineering community) is more important than other activity.

Proposed / discussed metrics

  • ☒ = not existing in korma
  • ☐ = in progress / exists outside korma / in korma but broken / etc
  • ☑ = exists in korma
  • ★ = KPIs to be promoted to WMF management

Code review and changes (Gerrit/Git)

★ Active Gerrit users per month☑ "Code Review users"; T86152
Active changeset authors per monthT97717: Active changeset *authors* and changeset *reviewers* per month; subgroup of above
Active changeset reviewers (setting ∓1/∓2, not comments-only) per monthT97717: Active changeset *authors* and changeset *reviewers* per month; subgroup of above
★ Changesets submitted per month☑ "Submitted" (though not sure if Patches or Changesets)
Changesets reviewed (∓1/∓2, not comments-only) per monthT97716: Gerrit changes reviewed per month (on scr.html); only "Merged" and "Abandoned" on korma's scr.html ?
Changesets waiting for review per month☑ "Pending" (though not sure if Patches or Changesets)
★ Median age of open changesets☑ "Age of open changesets (monthly snapshots)"; T97715
Users that have exercised ∓2 in Gerrit on a given month☑ "+2/-2 Votes"; T59038

(A patch contains of 1-∞ changesets)

Task management (Phabricator)

Anything as part of korma requires a T96238: Maniphest backend for Metrics Grimoire first. T96238 was finished in late June 2015 and in Q3 we will finetune Phabricator metrics to integrate and display in korma. For Q2, Git/Gerrit metrics are higher priority as we get our Phabricator KPIs already (via SQL) in the monthly email to wikitech-l at the beginning of every month.

Accounts per month that have performed at least one action☐ Currently exists only in monthly Phab stats email on wikitech-l but not as graph in korma
Tasks created per month☐ Currently exists only in monthly Phab stats email on wikitech-l but not as graph in korma
New accounts created per month☐ Currently exists only in monthly Phab stats email on wikitech-l but not as graph in korma

Broader level (not "basic" metrics): T28

IRC

Users per month who have sent at least one message☑ on http://korma.wmflabs.org/browser/irc.html (cf. T96371, T56230)

Mailing lists

Users per month who have sent at least one message☑ on http://korma.wmflabs.org/browser/mls.html

mediawiki.org editors

To be defined (see discussion in this task)

Potential followup ideas, not covered in this task

Related Objects

Aklapper created this task.Mar 31 2015, 4:25 PM
Aklapper updated the task description. (Show Details)
Aklapper raised the priority of this task from to Normal.
Aklapper claimed this task.
Aklapper added a subscriber: Aklapper.
Qgil added a subscriber: Qgil.Apr 6 2015, 6:31 PM

Should this task be part of ECT-April-2015 ?

Aklapper renamed this task from Ensure that most basic Community Metrics are in place to Ensure that most basic Community Metrics are in place and how they are presented.Apr 7 2015, 4:13 PM
Aklapper set Security to None.
Aklapper raised the priority of this task from Normal to High.Apr 13 2015, 1:31 PM

fyi, I was right about this task being a high priority goal for this quarter. ;)

The main piece of feedback we got from from Lila (not an exact quote):

You need to present top-level KPIs for ECT, for instance the number of developers engaged and contributing code, the number of patches submitted and accepted... this kind of data.

My reply was that yes, we know, this time we have only presented basic numbers about Phabricator usage, but one of the goals for the next quarter is T94578: Most basic Tech Community metrics are published and up to date and it is committed by the end of June, so we plan to present these metrics in the next quarterly review.

Qgil added a comment.Apr 16 2015, 4:12 PM

Fair. I have no idea what the resourcing for this will be, with Erik's departure, I'm afraid :(

Well, it should not be rocket science to agree on a v0.1 of these basic metrics and ask Bitergia to make an estimate. This is what I have requested to @Aklapper, and your help @Ironholds would be welcome. In fact, I think we could get this 0.1v giving us a week to add metrics to the description of this task, and then have a hangout with whoever has been active in this task.

What do you think? Should we start counting now and schedule a hangout for next Thursday 23?

I'd be happy to help, but the 23rd is straight after I (probably? It's...not entirely clear) shift to working for a new team with different priorities and needs and focus, so I have literally no idea what I'll be working on.

Qgil updated the task description. (Show Details)Apr 16 2015, 4:26 PM
Qgil added a comment.Apr 16 2015, 7:03 PM

@Ironhold, then you could dump your thoughts here before moving to the new team, and we will continue from there. :)

Aklapper updated the task description. (Show Details)Apr 20 2015, 11:14 AM
Aklapper updated the task description. (Show Details)Apr 20 2015, 12:06 PM

Edited the task description.
Need help to make that list way shorter. I've marked items that don't convince me with a "(?)" prefix.
Please be bold and remove items.

Aklapper updated the task description. (Show Details)Apr 20 2015, 12:09 PM
Qgil added a comment.Apr 20 2015, 12:26 PM

What about mediawiki.org editors?

  • New editors (editors reaching their 10th edit on a given month).
  • Active editors (editors with 5 edits or more on a given month).
  • Very active editors (editors with 100 edits or more on a given month).

These definitions are standard across Wikimedia, and I guess there is a tool somewhere gathering this data.

Aklapper updated the task description. (Show Details)Apr 21 2015, 3:07 PM
Aklapper updated the task description. (Show Details)
Aklapper updated the task description. (Show Details)Apr 21 2015, 3:11 PM
Aklapper updated the task description. (Show Details)Apr 21 2015, 3:15 PM
Aklapper updated the task description. (Show Details)Apr 21 2015, 3:42 PM
Qgil moved this task from Backlog to Doing on the ECT-April-2015 board.
Aklapper updated the task description. (Show Details)Apr 30 2015, 9:32 PM
Aklapper updated the task description. (Show Details)Apr 30 2015, 9:43 PM
Qgil added a comment.May 7 2015, 1:32 PM

What about

  • Users who contributed a first patch per month.

I think it is an important one, showing how are we doing in terms of outreach.

Then, I think it is important to distinguish in this metric and also in a couple more (see below) between users and changesets related to WMF and WMDE (paid by Wikimedia) and all the rest (volunteers and paid by third parties), because this shows how is the open source project doing beyond Wikimedia's capacity to hire developers.

  • Active changeset authors per month
  • Changesets submitted per month

I'm actually thinking that these numbers should be used to evaluate how good the Engineering Community team is doing. More new developers, and more contributors & patches without a WMF/WMDE affiliation means a more successful Engineering Community team. If we get less and less of those, then we are doing something wrong. If possible, I would like to show these numbers to our executive team and our community by the end of this quarter.

Qgil added a comment.May 21 2015, 9:19 AM

I was looking at the good summary of metrics at https://commons.wikimedia.org/w/index.php?title=File:Wikimedia_Foundation_Quarterly_Report,_FY_2014-15_Q3_(January-March).pdf&page=3. There is no mention about developers and our technical community in general.

We cannot blame them, because this task is still open. ;) However, which most basic Tech Community Metrics should be listed in that slide every quarter?

And related: which Tech Community Metrics should be featured in every monthly Metrics meeting?

Let's decide, and then I'll talk to Terry, who is driving both the quarter reviews and the metrics meetings.

Qgil added a comment.May 24 2015, 2:53 PM

After a casual conversation with Andre in Lyon, these could the KPIs to be promoted to Lila, Terry, and WMF management in general:

Aklapper updated the task description. (Show Details)May 29 2015, 8:55 AM
Aklapper added a project: ECT-June-2015.
Aklapper updated the task description. (Show Details)Jun 1 2015, 11:23 AM
Aklapper updated the task description. (Show Details)
Aklapper updated the task description. (Show Details)Jun 1 2015, 11:28 AM
Qgil added a comment.Jun 17 2015, 12:45 PM

We have to present the slides of the next quarterly review by June 30. This is about 8 working days from now. Just to give some perspective. :)

Qgil added a comment.Jun 22 2015, 7:10 AM

Related, although at this point this cannot be considered a blocker of this goal: T103292: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months

I'm realizing that we haven't identified as a key metric the number of successful code contributors per month (users that got their code merged in Wikimedia hosted repositories). We will have active changeset authors per month (T97717), but maybe it is useful to see compare how many of them actually have an impact on our repositories on a monthly basis? Something to be considered. We have the data anyway (although we need to check how reliable it is), so not that urgent/important.

Qgil renamed this task from Ensure that most basic Community Metrics are in place and how they are presented to Ensure that most basic Tech Community Metrics are in place.Jun 25 2015, 12:03 PM
Qgil renamed this task from Ensure that most basic Tech Community Metrics are in place to Most basic Tech Community metrics are published and up to date.Jun 25 2015, 12:06 PM
Aklapper updated the task description. (Show Details)Jun 25 2015, 6:57 PM
Aklapper updated the task description. (Show Details)Jun 25 2015, 7:15 PM
Aklapper updated the task description. (Show Details)
Aklapper updated the task description. (Show Details)Jun 25 2015, 7:19 PM
Aklapper updated the task description. (Show Details)Jun 25 2015, 7:21 PM
Qgil added a comment.EditedJun 27 2015, 8:20 AM

Hm, now http://korma.wmflabs.org/browser/gerrit_review_queue.html doesn't quite load in my laptop (just some frames and one graph). Can you try, please?

Aklapper updated the task description. (Show Details)Jun 30 2015, 3:58 PM
Aklapper closed this task as Resolved.

All data we wanted (as defined in the task description) is available in public places now, so I'm closing this task as resolved.
Thanks everybody for input, code, discussion, support.

Further discussing scope, visualization (e.g. T100978) or reliability (e.g. T103292) to be handled in dedicated followup tasks.