Page MenuHomePhabricator

Newcomer tasks: experiment analysis
Closed, ResolvedPublic

Description

This is for analyzing the A/B tests for activation, retention, and other KPIs that are run as part of the newcomer tasks project.

In particular, we want to analyze the following:

  • Activation
  • Activation (counting only constructive edits)
  • Retention
  • Retention (counting only constructive edits)
  • Edit volume
  • Constructive edits
  • Count of number of highly active users
  • Number of thanks received by users

We want to look at differences between wikis and between platforms. In terms of namespaces, we can look at these three groupings separately:

  • All namespaces
  • Article and Article Talk
  • All namespaces besides Article and Article Talk

Event Timeline

Moving to Q2, feel free to move @MMiller_WMF

MMiller_WMF updated the task description. (Show Details)
MMiller_WMF edited projects, added Growth-Team (Current Sprint); removed Growth-Team.

This experiment was deployed 3.5 months ago. It is ready for analysis.

LGoto triaged this task as Medium priority.Apr 13 2020, 4:44 PM
nettrom_WMF renamed this task from Newcomer tasks: experiment to Newcomer tasks: experiment analysis.Apr 21 2020, 8:00 PM

Morten has started this analysis, it's partially done, but moved back into next 2 weeks because https://phabricator.wikimedia.org/T250120 was prioritized above this.

@nettrom_WMF and I have set a date for this of Friday, June 26.

The draft is now with me for my review and writing of executive summary.

@MMiller_WMF, @nettrom_WMF: The Due Date set for this open task was more than two months ago.
Can you please either update or reset the Due Date (by clicking Edit Task)? Thanks.

nettrom_WMF changed Due Date from Jun 26 2020, 7:00 PM to Sep 25 2020, 7:00 PM.Sep 14 2020, 4:26 PM

The Due Date set for this open task was two months ago. How long does "Sign-off" take?

The Due Date set for this open task was two months ago. How long does "Sign-off" take?

Hah, thanks for the reminder about that, @Aklapper! I've removed the due date and recategorized this task a bit while we continue to iron out the details.

Momentarily blocked by T266982; Morten & Marshall want a deeper understanding of productivity to incorporate into the analysis. When T266982 is resolved, this one will be resolved.

Marshall put together a draft report ready to be published on wiki, once the productivity analysis is added.

This analysis is almost complete. Still left to do is to update the googledoc with the new edit volume estimates, move it into the Growth team's shared drive, and remove its draft status.

The googledoc has been updated and moved to the Growth team's shared drive. The findings are also summarized in this report on-wiki.