Page MenuHomePhabricator

Help panel: hierarchical analysis
Closed, ResolvedPublic


Now that we have developed a hierarchical method to analyze experiments across wikis, we want to use it to analyze the help panel experiments, using data from Czech, Korean, and Vietnamese Wikipedias. The main metrics we want to look at are:

  • Activation
  • Retention
  • Edit count/volume

A nice-to-have metric would be "non-reverted edits", or some other version of measuring productive edits.

Event Timeline

nettrom_WMF triaged this task as Medium priority.Jul 8 2019, 5:03 PM
nettrom_WMF moved this task from Triage to Next Up on the Product-Analytics board.
nettrom_WMF edited projects, added Growth-Team (Current Sprint); removed Growth-Team.

I've completed gathering data for this analysis using the 2019-06 MediaWiki history snapshot. Moving it to "In Progress" as I'll start doing the actual analysis.

@nettrom_WMF -- per our conversation, I've added "edit count/volume" as one of the KPIs we want to look at, in addition to activation and retention. I also added "non-reverted edits" as a something that would be interesting, but not a high priority.

MMiller_WMF set Due Date to Aug 23 2019, 7:00 AM.
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptAug 10 2019, 12:48 AM

Analysis piece is done, next up is reporting & sharing with stakeholder

We have a set of slides for sharing this with stakeholders, reassigning to @MMiller_WMF for review.

We have completed this analysis, and I posted some of the results in brief in this update (and will post some more):

Now reassigning this back to @nettrom_WMF and putting it into Ready for Development because the final step is to write a fuller report on wiki. We need to decide when and how to do this.

Moving this to current quarter to be done together with other documentation tasks.

We've completed the analysis but not gotten around to publishing it on-wiki. There were not any substantial findings, instead the team learned a lot about the challenge of running experiments with features that are used by a relatively small percentage of users. Since then, we've repurposed the Help Panel to support Newcomer Tasks, where we've seen a larger impact. It doesn't seem like a good use of time and resources to work on publishing these non-results at this point, so I'm closing this task as resolved.