Page MenuHomePhabricator

Variant tests: revisit edit completion for A vs. B
Closed, ResolvedPublic

Description

When we analyzed Variant A vs. B in T238888, we looked at three metrics: interaction rate, navigation rate, and task click rate. At the time, we did not have the sample size to look at the rate of completing a suggested edit, which is the outcome we ultimately want to affect.

The results from the analysis we did materially informed our designs for T246533: Variant tests: "initiation part 2" test (C vs. D), and we intended to quickly build and deploy that experiment. But that experiment has become delayed, and in the meantime, we want to give newcomers A or B, whichever is better.

Therefore, we want to quickly look into whether Variant A or B has a higher rate of newcomers completing a suggested edit. Now that it is many weeks after the initial experiment, we may have the sample size to do this.

Event Timeline

MMiller_WMF updated the task description. (Show Details)May 28 2020, 7:02 PM
nettrom_WMF triaged this task as High priority.May 28 2020, 10:43 PM

The first pass of this analysis is now complete. For the data gathering, I used the same approach as for the previous analysis, except I could gather more data and set 2020-05-15 as the end date. As before, known test accounts, control group users, and users who turned the Homepage on or off were excluded. I used data from the Data Lake to count number of tagged edits made by each user over the same 14 day period after registration that we used to measure Homepage interactions in the previous analysis.

Just like we did in the previous analysis, we find a strong interaction effect between variant A/B and whether the user registered on the desktop or mobile site. This means our results need to split up by desktop/mobile, and can be summarized as follows:

  1. On the desktop site, there is no significant difference in probability of making a tagged edit between the two variants.
  2. On the mobile site, there is a significantly lower probability of making a tagged edit if the user was in Variant B.

We also find that there's overall a significantly higher probability of making a tagged edit if the user is on mobile. I speculate that this might be due to the mobile overlay that asks users to create an account if they're trying to edit without being logged in on mobile, as we've previously seen higher registration rates from an editing context on mobile.

Conclusion: We find conclusions that conflict with our findings from the earlier analysis, where Variant B was found to clearly result in higher interactions on the desktop platform and no change on mobile. It might be worth digging into the data to try to understand why we have a significant drop on mobile in Variant B, or if that's not worth it, probably give all users variant A.

Handing it off to @MMiller_WMF for review.

BTW, for reference, the code for data gathering is in this notebook, and the regression model analysis is in this notebook.

MMiller_WMF closed this task as Resolved.Jun 3 2020, 11:19 PM