Page MenuHomePhabricator

Welcome emails: experiment analysis
Closed, ResolvedPublic

Description

We want to analyze this campaign to understand whether sending out emails to newcomers had an effect.

Context:

  • The campaign was run on Spanish Wikipedia
  • It started on 2022-05-10 13:50:02 UTC (ref T307844#7917482)
  • It'll end on 2022-07-01 00:00:00 UTC (in other words, we'll not use any registrations from then on)
  • During the experiment, users can opt-in to receiving emails when responding to the Welcome Survey
  • Users are randomly assigned to a treatment or control group, and this group assignment is stored in the Welcome Survey response.

We are interested in understanding:

  1. Differences between users who opt-in to receiving the emails and those who don't.
  2. Differences between users who got the email and those who didn't.

These differences might be along the lines of:

  • Activation (editing within 24 hours of registration)
  • Retention, in the broadest sense as the Growth team has a specific definition of retention that requires activation.
  • Productivity (number of edits made)

Exploratory analysis might provide good indications of what metrics are helpful and will provide meaningful insight into how these groups differ.

Event Timeline

mpopov triaged this task as Medium priority.Jul 5 2022, 5:14 PM
mpopov moved this task from Triage to Current Quarter on the Product-Analytics board.

Moving this to the sign-off column on the PA board for now, while I await review of the draft report.

Urbanecm_WMF changed the task status from Open to In Progress.Sep 29 2022, 10:06 AM

I checked with @EdErhart-WMF and he confirmed that there is no further analysis needed, so I believe we can consider this task Resolved.

Reopening this as @EdErhart-WMF and I want to get the analysis results up on MediaWiki.org for easy referencing. This will most likely end up as a separate subpage of the project page, with a paragraph about it on the project page.