Page MenuHomePhabricator

Newcomer tasks: NEWTEA revisited
Closed, ResolvedPublic

Description

The analysis of the Newcomer Tasks experiment (T230174) was complicated by the Variant A/B test (T238888). It also did not incorporate important developments such as Guidance. We therefore want to revisit it using a dataset of registrations from a period where Guidance was available and the feature set was stable. In this analysis we'll focus on the four main questions:

  1. Activation
  2. Retention
  3. Edit volume
  4. Revert rate

For all of these, we want to examine effects overall and for the Article & Article talk namespaces specifically. For edit volume and revert rate, we'll also want to limit it to the same time periods used for activation and retention, just like we did in the previous analysis.

The main goal of this analysis is to get reasonable estimates of the effect of the Homepage with Guidance during a period where features were stable. hopefully confirming the findings from NEWTEA.

Event Timeline

LGoto triaged this task as Medium priority.Jan 5 2021, 6:09 PM
LGoto moved this task from Triage to Current Quarter on the Product-Analytics board.
nettrom_WMF changed the task status from Open to Stalled.Sep 29 2021, 9:07 PM

Went through the results of analyzing the first subset of registrations with Marshall yesterday, and we decided on a narrowly scoped follow up to understand activation and retention for the remaining subsets. This work is on hold until T292103 and possibly other subtasks of T289795 are completed.

nettrom_WMF changed the task status from Stalled to In Progress.Nov 9 2021, 6:04 PM

I'm working on revising our draft analysis report and putting it on mediawiki.org. Once that is done this task will be resolved.

I'm working on revising our draft analysis report and putting it on mediawiki.org. Once that is done this task will be resolved.

What is your plan there? Publishing a new page, or adding the newest information to mw:NEWTEA?

Both options have their pros and cons: a separate page means a separate report while mixing both makes it easier to compare the results.

What is your plan there? Publishing a new page, or adding the newest information to mw:NEWTEA?

Both options have their pros and cons: a separate page means a separate report while mixing both makes it easier to compare the results.

I've so far worked on a draft in my user space that's a separate page. Your excellent question got me thinking about this, though! I think it might be better to combine it all into one page so it's easy to see the full results without having to switch between the pages. Both analyses are very similar since we haven't made significant changes to our methodology, we're focusing on the same key metrics, and some of the results are the same. Once I have a first draft of all the results sections, I'll work on changing that draft by pulling in the same sections from mw:NEWTEA so we can later have it all on one page.

nettrom_WMF changed the task status from In Progress to Stalled.Mar 24 2022, 4:00 PM
nettrom_WMF moved this task from Doing to Blocked on the Product-Analytics (Kanban) board.

Changing the status of this as it'll be on pause for a few weeks. Once I pick it up again the plan is to do as I described in T270786#7776009, that I'll edit the results so they fit into the current mw:NEWTEA page and we have everything in one place. It'll probably mean we move the page as well, because the "November 2020" part of the title will no longer be correct.

nettrom_WMF changed the task status from Stalled to In Progress.Apr 25 2022, 4:53 PM
nettrom_WMF moved this task from Blocked to Doing on the Product-Analytics (Kanban) board.

Moving this off the PA Kanban and Growth sprint boards as this work has to wait for a bit until I can pick it up again. It's important that we make these results available to the community, and we'll need to schedule this work together with other things for the current quarter.

Pppery changed the task status from In Progress to Open.Nov 11 2024, 5:24 AM
Pppery moved this task from Upcoming Work to Triaged on the Growth-Team board.

Resolving this task as the work has been published in the Newcomer Homepage paper, presented at CSCW 2023. We also presented this work in the May 2024 Research Showcase.