Page MenuHomePhabricator

Investigate how to accomplish A/B testing in the iOS app
Closed, ResolvedPublicSpike


Epic story: As a product manager I want to be able to run a variant test and understand the impact on user behavior.

Must have:

  • ability randomly assign users to a variant/split test experience of "control" group
  • ability to define a % of users who will be in each group
  • ability to track events based on the pool a user is assigned to

Originally we had aimed to have server side control of test parameters and pooling, for now we'll code pool assignment on the client.

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptJun 15 2020, 6:05 PM
LGoto triaged this task as Medium priority.Jun 15 2020, 6:37 PM
LGoto removed a project: Epic.
LGoto moved this task from Needs Triage to Engineering Backlog on the Wikipedia-iOS-App-Backlog board.

We also might be able to accomplish A/B testing through the new Modern Event Platform stuff. With it you can set up different sampling rates for different experiments. We should explore this first since we need to review their PR anyways.

JMinor raised the priority of this task from Medium to High.Jul 9 2020, 7:30 PM

Per our meeting, I'll write up a basic story/acceptance and we'll see how that fits with MEP.

JMinor updated the task description. (Show Details)
JMinor added a subscriber: JMinor.

Notes: In Android, on app side, when first time user gets to feature there is logic that picks a random number even/odd, user falls into that group. Once the user is in a group, a field is tagged with random number that indicates group. Tag gets persisted for lifetime of app.

Need to add column to table for test group designation. Field should be string, can hold multiple values.

More notes on how Android does this:

To decide which group to put the user into, they take a random number modulo 2, and persist it for the lifetime of the install. They just send one event when the user's group membership is first determined, since it's all about how many unique installs fall into each group. Also note that the same schema can be used for multiple a/b tests at once. One test could send events with test groups of "featureA_groupA" / "featureA_groupB", and another could send "featureB_groupA" / "featureB_groupB". Group assignment happens when that feature is actually used. i.e. when the app itself checks which group the user belongs to, it lazily determines membership, if it hasn't been done yet.

@SNowick_WMF let us know if there's anything left on your end to do with this. If the schema is ready then I think we can move this task to Ready for Dev on our end.

We're all set until we actually need to run the test.

Change 634080 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/doxygen@debian/buster-wikimedia] Link to static libclang

Change 634080 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/doxygen@debian/buster-wikimedia] Link to static libclang

Sorry I have meant to link to T254465