The first pass of this analysis is now complete. For the data gathering, I used the same approach as for the previous analysis, except I could gather more data and set 2020-05-15 as the end date. As before, known test accounts, control group users, and users who turned the Homepage on or off were excluded. I used data from the Data Lake to count number of tagged edits made by each user over the same 14 day period after registration that we used to measure Homepage interactions in the previous analysis.

Just like we did in the previous analysis, we find a strong interaction effect between variant A/B and whether the user registered on the desktop or mobile site. This means our results need to split up by desktop/mobile, and can be summarized as follows:

1. On the desktop site, there is no significant difference in probability of making a tagged edit between the two variants.
2. On the mobile site, there is a significantly lower probability of making a tagged edit if the user was in Variant B.

We also find that there's overall a significantly higher probability of making a tagged edit if the user is on mobile. I speculate that this might be due to the mobile overlay that asks users to create an account if they're trying to edit without being logged in on mobile, as we've previously seen higher registration rates from an editing context on mobile.

Conclusion: We find conclusions that conflict with our findings from the earlier analysis, where Variant B was found to clearly result in higher interactions on the desktop platform and no change on mobile. It might be worth digging into the data to try to understand why we have a significant drop on mobile in Variant B, or if that's not worth it, probably give all users variant A.

Handing it off to @MMiller_WMF for review.

BTW, for reference, the code for data gathering is in this notebook, and the regression model analysis is in this notebook.

Thank you, @nettrom_WMF. The team has decided to switch all newcomers to Variant A as we work on T246533: Variant tests: "initiation part 2" test (C vs. D). That switch will happen in T254413: Variant tests: switch all newcomers to Variant A.

The results of this task are now here in our slides about the Variant A vs. B test: https://docs.google.com/presentation/d/1hvDHNTLZ9Lr7CjHqtYOUjSU2VZtN5AtTfh2Xnstu56s/edit#slide=id.g80a9ddb958_1_0