The Reader Growth team will make extensive use of XLab, the new WMF analytics platform, over the course of the next year as we launch various experimental features and test their effectiveness at retaining logged-out readers.
One of the first things we need to do here is establish a baseline for the new "retention" metric we will rely on in the future to measure the success of various features. Retention here can be defined as: Percentage of users who revisit wiki after their initial visit over a specific period (probably a 30-day window).
From @jwang
This could be a great project for onboarding to the Experiment Platform, as it involves an AA test—which is simpler than an A/B test—and only requires basic user interaction tracking (specifically, page load events).
For reference, here is a previous AA test implemented by Sam: T393918. FYI, to help evaluate the expected workload. Here is the product requirements for additional context. cc
Resources
Next Steps
In the future XLab may provide a more automated workflow for these kinds of tests. But for now we still need to do some manual setup. Here is what needs to happen to launch this A/A test. Each of these steps should be tracked in a dedicated sub-task (this might not be necessary in the future as we get more comfortable with these tools, but for the first one we do as a team let's follow each step individually).
- T399347 Finalize an instrumentation spec for the test
- T399482 Set up instrumentation for this experiment in the WikimediaEvents codebase
- T399484 Set up a "smoke test" for this experiment on TestWiki
- T399486 Enable the experiment in production via a config patch
- T399487 Disable the experiment in production via a config patch, after about 1 month's time