Page MenuHomePhabricator

Review key performance indicators (KPIs) for Community Liaison and Developer Relations
Closed, ResolvedPublic

Description

These are the metrics being proposed in our draft for T124019: Define the Technical Collaboration annual plan FY2016-17:

Goal: Community liaison support to WMF Product teams around new initiatives and ongoing complex projects

Goal: Consolidation of the Product Development Process as a systematic approach to community engagement by WMF Product teams, corresponded by the Wikimedia communities.

  • Number of major Product discussions within the PDP vs out of process.
  • Global metric: Did your work increase the motivation of contributors, and how do you know?

Goal: Connect free software developers with the Community Wishlist maintained by the Community Tech team.

These metrics are under review just like the rest of the Annual Plan proposal. We should start working on them during the next quarter (April - June) in order to have a baseline by the beginning of the new fiscal year.

Previous description

Let's discuss Community Liaisons and Developer Relations quarterly goals, with the goal of having satisfactory metrics by our next quarterly review in April.

Community Liaisons: are we happy about the metrics being collected currently?

Developer Relations: the KPI about users of our APIs is simply not working (not technically working, not aligned with our actual work still today).

When thinking about the right KPIs, we need to consider the direction of these teams in the WMF Strategy and Annual Plan.

Event Timeline

Qgil created this task.Jan 20 2016, 10:54 AM
Qgil updated the task description. (Show Details)
Qgil raised the priority of this task from to Normal.
Qgil claimed this task.
Qgil added subscribers: Keegan, Trizek-WMF, Quiddity and 8 others.
Elitre updated the task description. (Show Details)Jan 20 2016, 11:49 AM
Elitre set Security to None.
Samtar added a subscriber: Samtar.Jan 20 2016, 5:11 PM
Qgil removed Qgil as the assignee of this task.EditedJan 22 2016, 6:41 PM

Actually I'm not the best assignee for this task. De-assigning.

We need to look first at the CL KPIs, and we can use T124041 as a real use case for new KPIs. A first draft is expected by Feb 28... @Rdicerb, so you want to drive this task?

There is an interesting reference, which is the global metrics that by default the FDC requests for any organization they fund:

Global Metrics					
1. Number of active editors involved					
2. Number of new registered users					
3. Number of individuals involved					
4. Number of new images/media added to Wikimedia article pages					
5. Number of articles added or improved on Wikimedia projects					
6. Number of bytes added to and/or deleted from Wikimedia projects					
7. Learning question: Did your work increase the motivation of contributors, and how do you know?

Can we extract or extrapolate any KPIs taking this list as a reference?

Another reference are the main themes of the strategy: Reach, Community, Knowledge. Can we define any KPIs measuring the impact of our work against these areas?

Marking T116370: Community health survey for Wikimedia technical spaces tentatively as a blocker, just in case such survey would have anything to do with our KPIs.

The metrics that CLs are collecting are not ideal, as they measure community activity rather than the team's. The challenge faced is that we don't have easy tools for measuring staff activity. One example of possible KPI is "time to first response on Talk page" - that isn't easily measurable without a script (that I'm aware of).

I'll continue poking at this in the next few weeks.

Qgil added a comment.Jan 29 2016, 10:21 AM

Just copy&pasting some notes we had in some internal doc:

  • Number of volunteer developers and contributors involved in software projects led by the Wikimedia Foundation.
  • Number of active tech ambassadors and wiki projects covered by them.
  • Number of developers contributing their first 1 / 5 / 10 patches to Wikimedia projects.
  • Number of Community Tech backlog projects completed by others.

Let me paste our stated goals here:

  1. We want to measure results, not activity. However, we recognize that results are not fully under our control. For example, our results are affected by the actions of other WMF teams and by the phase that each product is in.
  2. We want something that is meaningful, even if it requires thoughtful interpretation.
  3. We want something that can be measured now, with no new tools.
  4. We want to change these later, if we find better indicators or get better tools.
  5. At this time, we are focusing on our community-facing roles, rather than our staff-facing roles. Consequently, our KPI will only measure part of our work.

If you're aware of anything that changed since our first iteration (like new tools making it possible for us to measure different things now), or any teams which have been able to prove (not just "correlate") that the impact they see in community metrics is actually caused by work they and they alone did (which TBH is not what I think when I read KPIs for teams in Product), then LMK?

Qgil added a comment.Feb 3 2016, 11:51 PM

T124041: Technical Collaboration narratives and budget for core work are still missing KPIs for the work that Community Liaisons currently do supporting WMF teams directly. Since that is a draft, should we list the KPIs that you are currently using? While we define better ones (if we find better ones).

For Core work, consistency around the Liaisons' work in product development (Project/initiative 1) will add cohesion. The pages listed here are not inclusive or consistent with the engagement that's actually happening across the products.

In talking with Quim yesterday, we keep coming to "the one thing" that can be measured. Given the nature of the work that the team does, it's a bit complex to find one *place* where something can be measured, but on thing can perhaps be delineated. I'm doing some reading this week and next and will try to throw a few ideas out there.

Rdicerb raised the priority of this task from Normal to High.Feb 10 2016, 10:32 PM

"Time to first reply" is a customer service metric, not a community management one. Also, the best proof of my success is when the local community members can answer each other's questions. That means that I've built a sub-community with knowledge, arranged for useful and organized documentation, etc.

In terms of "supporting WMF teams directly", we could always just ask the teams.

Our current KPIs don't cover our interactions with staff at all.

@Whatamidoing-WMF - you are raising a good point around the customer service metric, thank you for always bringing your ideas to this! There's more information the KPI page. Might be better to hold the conversation there?

Rdicerb added a comment.EditedFeb 24 2016, 7:10 PM

In a couple of session some team members gathered to recommend a new CL KPI. The current thinking is that we may count "major conversations" that the team participates in for individual products, but the definition of what constitutes a major conversation needs to be clarified.

Rdicerb reassigned this task from Rdicerb to Qgil.Mar 2 2016, 6:40 PM

Back to you, Quim, as there is the Dev Rel element to this task that I think yourself, Rachel F or Andre will have a better answer to.

The current thinking is that we do a flat count for Q4, and while doing so consider ways of qualifying the discussions had, with a proposal of how to qualify these conversations beginning Q1'16/17.

MZMcBride renamed this task from Review KPIs for Community Liaison and Developer Relations to Review key performance indicators (KPIs) for Community Liaison and Developer Relations.Mar 14 2016, 11:43 PM
MZMcBride added a subscriber: MZMcBride.
Qgil updated the task description. (Show Details)Mar 17 2016, 11:38 AM
Qgil lowered the priority of this task from High to Normal.Mar 17 2016, 11:41 AM

Now the description of this task contains the KPIs being proposed in our annual plan.

These metrics are under review just like the rest of the Annual Plan proposal. We should start working on them during the next quarter (April - June) in order to have a baseline by the beginning of the new fiscal year.

I'm lowering priority only because my work on this task between now and the approval of these KPIs is likely to be reactive to feedback, less pushing actively.

Qgil closed this task as Resolved.Apr 2 2016, 1:29 PM

The Technical Collaboration KPIs are now defined at https://meta.wikimedia.org/wiki/Technical_Collaboration/Strategy#Processes

The WMF Annual Plan FY2016-17 draft also contains measurable goals for our team:

With this results, this task can be resolved. If we need changes or more detail for quarterly reviews etc, these can be proposed and discussed in new tasks.