Epic story: As a product manager I want to be able to run a variant test and understand the impact on user behavior.
Must have:
Originally we had aimed to have server side control of test parameters and pooling, for now we'll code pool assignment on the client.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T241252 Editing Awareness and Trust Experiments | |||
| Resolved | Spike | SNowick_WMF | T255465 Investigate how to accomplish A/B testing in the iOS app | ||
| Resolved | Spike | SNowick_WMF | T259659 Update schema to add test_group column to MobileWikiAppiOSUserHistory |
We also might be able to accomplish A/B testing through the new Modern Event Platform stuff. With it you can set up different sampling rates for different experiments. We should explore this first since we need to review their PR anyways.
Per our meeting, I'll write up a basic story/acceptance and we'll see how that fits with MEP.
Notes: In Android, on app side, when first time user gets to feature there is logic that picks a random number even/odd, user falls into that group. Once the user is in a group, a field is tagged with random number that indicates group. Tag gets persisted for lifetime of app.
Need to add column to table https://meta.wikimedia.org/wiki/Schema:MobileWikiAppiOSUserHistory for test group designation. Field should be string, can hold multiple values.
More notes on how Android does this:
To decide which group to put the user into, they take a random number modulo 2, and persist it for the lifetime of the install. They just send one event when the user's group membership is first determined, since it's all about how many unique installs fall into each group. Also note that the same schema can be used for multiple a/b tests at once. One test could send events with test groups of "featureA_groupA" / "featureA_groupB", and another could send "featureB_groupA" / "featureB_groupB". Group assignment happens when that feature is actually used. i.e. when the app itself checks which group the user belongs to, it lazily determines membership, if it hasn't been done yet.
@SNowick_WMF let us know if there's anything left on your end to do with this. If the schema is ready then I think we can move this task to Ready for Dev on our end.
Change 634080 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/doxygen@debian/buster-wikimedia] Link to static libclang