Page MenuHomePhabricator

Investigate and recommend a core metric for editor retention
Closed, ResolvedPublic

Description

Conduct a metric feasibility investigation to evaluate configurations of a core metric focused on "editor retention."

See more detail in project plan

Details

Due Date
Jun 19 2025, 4:00 AM

Event Timeline

OSefu-WMF triaged this task as High priority.
nshahquinn-wmf renamed this task from Metric Feasibility Investigation - Editor Retention Core Metric to Investigate and recommend a core metric for editor retention.Apr 24 2025, 12:05 AM

Last week, I:

  • Created a rough week-by-week project schedule
  • Brainstormed high-level retention metric patterns
  • Had exploratory conversations with Hamid and Mikhail

This week, I:

  • Chose and wrote up 2 straw dog metrics (second-month active editor retention and second-week edit retention)
  • Identified key decision points for metric construction
  • Had a lively check-in meeting with key stakeholders
  • Decided that the metric should be a count of retained editors rather than a retention rate
  • Planned the metric prototyping with Hamid
nshahquinn-wmf changed Due Date from May 30 2025, 4:00 AM to Jun 12 2025, 4:00 AM.May 29 2025, 4:33 PM
nshahquinn-wmf changed Due Date from Jun 12 2025, 4:00 AM to Jun 19 2025, 4:00 AM.Jun 6 2025, 9:28 PM

Last week, I:

  • Calculated and analyzed the second-week edit repeaters candidate metric

This week, I:

  • Calculated the full history of the second-month retained active editors candidate metric and compared it with active editors
  • Had a second check-in meeting and several 1:1s discussions with contributors
  • Facilitated a decision to move forward with second-month retained active editors
  • Facilitated a follow-up decision to lower the qualification to 1 edit, resulting in second-month active editors
  • Calculated and compared second-month retained editors

This week, I:

  • Drafted the decision brief (WMF only) and shared it with project contributors for feedback
  • Rewrote queries for the prototype metrics to group data based on the end of the retention window (which will be helpful for the expanded retention framework)
  • Created the first set of polished charts for the decision brief

Last week, I:

  • Wrote the decision brief section on segmentations and indicator metrics
  • Finished creating polished graphs for the decision brief

The brief is now under review by the decision-makers.

This week, I:

  • Responded to lots of comments on the brief

Since the feedback on the recommendation has been generally positive, my work here seems to be done and I expect to close this task next week.

This week, I:

Last week, I:

  • Finished updating the brief in response to Kate and Marshall’s concerns around indicator metrics, segmentation, and the retention window
  • Met with Kate and Marshall for a detailed discussion of the responses, which went very well.

I think we're on track to have the decision approved by the end of this week!

This week, I:

  • Did (hopefully) final cleanup and tweaks on the decision brief
  • Had another review meeting with Kate and Marshall

Kate has signed off, but Marshall will review it with Selena before he does so. That puts us on track for approval in the next week or two.

Selena and Marshall have approved!