Page MenuHomePhabricator

Newcomer tasks: impact on active editors metrics
Closed, ResolvedPublic

Description

In preparing for the Q2 tuning sessions, @cchen and @kzimmerman found a strong YoY growth in active editors. Many of the wikis with Growth features show above average growth. We're wondering how much of that can be attributed to the Growth features.

The plan is to grab the dataset of users for NEWTEA revisited, and then investigate how the proportion of new and returning active editors compares between the Control and Growth features group. We can then reuse that dataset of users for the remaining NEWTEA revisited analysis.

Event Timeline

This has been completed, here's the update I gave to Connie, Kate, and Marshall in preparation for the Q2 Tuning Session:

Hey folks, I’ve gathered some data and crunched some numbers. What I end up finding is that there appears to be a small positive difference in favour of the Growth features for New Active Editors, but that this difference is not statistically significant. We’re talking about a kind of “needle in a haystack” problem, looking at registrations on Growth wikis from mid-June through August, 5.63% of the Control group registrations reached “new active editor” status, while 5.92% of the Growth features group did (that 0.29pp increase is +5.0%). For returning active editors, looking at the month after registration, the difference is basically zero.

I repeated this using the dataset from the Variant test [edit: that's T264831] (because it’s also random and has basically the same criteria for inclusion), and find the same results. I think it’s worth noting that the proportions of New Active Editors is roughly similar in both datasets: 5.67% vs 5.97% Control/Growth (0.3pp for +5.2%).

I’ve also thrown hierarchical logistic regressions at the problem to account for differences between wikis, and find again suggestions of no significant difference. There are indications of variance between wikis, but we’re challenged by the effect sizes being small. This does suggest that looking at this across all wikis might cancel things out, but I’m reluctant to do wiki-specific analyses without having solid selection criteria.
One thing I do find with the regressions is that there’s meaningful variation between wikis when it comes to the effect of desktop/mobile registrations on probability of reaching Active Editors status (both New and Returning). There’s always lower likelihood on mobile, but the magnitude varies strongly (e.g. Arabic has a fairly small effect, Czech has a fairly large effect).