Page MenuHomePhabricator

Define appropriate retention metrics for Event Tool
Closed, ResolvedPublic

Description

The Problem

CommTech is building an Event Tool for organizers of things like editathons. Metrics are a big focus. Among the things these users want to track is retention: What % of new users introduced to the wikis via an editathon continue to edit?

Sati has requested that we track retention over an extended period: she’s requesting 7-day, 30-day, 60-day, 6 month, 1 year. The general idea makes sense, but the problem is that no one knows how to define any of those figures, or if we should be using somewhat different intervals.

We’d like to use standard definitions if we can, so that we can compare event retention to, say, general retention on the relevant wiki.

What we need
  • Descriptions of standard definitions for at least a small set of short- and long-term retention metrics.
    • Please be clear about how the different metrics are related (e.g., are all users who are retained in the third month also necessarily retained in the second? Or does the second-month definition include only those who make an edit between 30 and 60 days?)
  • If available, links to existing queries that are being used to derive some of these figures.

If no standards have been decided on, then a best-guess solution will have to do.

Deadline
  • It would be great to have best-guess definitions for 2 short-term (e.g., 1-week and 2nd month) and one long-term (e.g., 6 months?) by the beginning of October and
  • Final decisions for those by the end of October?

Event Timeline

This is essentially the same as T201501: Develop a framework for measuring user retention. In other words, I think standardizing these metrics would be great 😁

MBinder_WMF triaged this task as Normal priority.Aug 30 2018, 8:21 PM
MBinder_WMF moved this task from Triage to Next Up on the Product-Analytics board.
Abit added a comment.EditedAug 30 2018, 10:15 PM

Sati and I also did further research on what metrics are important to program organizers a year or two after that, leading up to the grant metrics tool. @Shouston_WMF, do you remember where any of that research ended up?
EDIT: Ha, I see you are part of planning this and probably already thinking of the research we did. Carry on!

nettrom_WMF moved this task from Next Up to Doing on the Product-Analytics board.Oct 5 2018, 3:49 PM

Thank you for your patience while I’ve been working on this! There’s been quite a bit of coordination behind the scenes that slowed the process down.

The way I look at this, after digging into some of the community discussions and looking at what this task asks for, is that retention for programs/events can be split into three categories:

  • short-term on-wiki retention
  • long-term on-wiki retention
  • off-wiki retention (participation in other programs/events)

I am unsure whether the latter applies here? For now, I’ll focus on the first two.

As far as I know, short-term editor retention is not used in a standardized way. The Growth Team is proposing to track two-week retention after an activation event (typically an account that edits on their first day), and are working on documenting their key metrics and how they came to decide on those. In short, they chose two weeks to allow for someone to return the weekend after registration and edit then (e.g. someone registers on Saturday and returns on Sunday eight days later).

I am unsure about what an appropriate definition of “short term” would be in programs/events. I see that 7-day retention is asked for, has that already been in use among program/event organizers? It would be great to learn a bit more about the rationale behind choosing 7 days, and how it would be useful for organizers, particularly in relation to long-term metrics.

For long-term retention, the standard metrics are rolling surviving new active editor and rolling recurring old active editor. This is calculated on a two-month basis. Based on the community discussion and the description in this task, it seems to be a poor fit for this purpose as programs and events appear to be strongly cohort-based.

Because tracking retention over a long period is requested, I propose that the tool uses a retention matrix to track retention in the 12 months following a program/event. See this blog post from Facebook Analytics for an example, scroll down to see the example weekly retention chart (I prefer to call it a matrix, I’ve also seen it referred to as a “graph”). It would track the cohort of users in a program/event and count them as active in a given month after the event if they edited in that month (we could use 30-days as a “month” for simplicity, and make the twelfth month 35 days if we want a 365-day year).

Two concerns are whether retention is cumulative, and that measuring small events might be problematic. With regards to retention being cumulative, e.g. that a user can only be retained in the third month if they’re also retained in the second, I propose that it should not be. Participation numbers in specific events might be low, making the numbers quickly reach zero if we require retention to be a funnel. As mentioned, measuring small programs/events might be problematic, either because we’re trying to measure “on average half a person”, and that the behaviour of a single participant can lead to large percentage swings. One thing to consider is the ability to aggregate results across multiple events.

The need for comparisons against benchmarks came up during the discussion, and is also mentioned here in the sense of comparisons against standard metrics. With regards to the proposed 12-month retention, there is no benchmark to compare against as that is not used elsewhere. You could measure and compare against the rolling surviving (new) editor metric, but I am unsure how meaningful that would be for a program/event organizer. Arguably, it would only be meaningful for the second month after the program/event. In short, one thing to consider is whether the benchmark is the “average program/event” (thus being tracked across all programs/events), or if there is a need for further investigation into what a meaningful on-wiki retention benchmark would be.

nettrom_WMF moved this task from Doing to Epics on the Product-Analytics board.Oct 18 2018, 8:32 PM
Restricted Application added a project: Community-Tech. · View Herald TranscriptApr 25 2019, 4:57 PM

@Magalia, tell what you need.

nettrom_WMF closed this task as Resolved.Jul 15 2019, 4:13 PM

As far as I know, the tool has been developed and deployed, so this task can be closed.