Task
Validate event data for AB test instrumentation
- Beta - DONE spreadsheet
- Prod
Documentation
iOS Home Feed Data Instrumentation Planning Deck
Validate event data for AB test instrumentation
iOS Home Feed Data Instrumentation Planning Deck
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T427526 [Epic] Scale Home Feed to iOS (Explore Feed Revamp) | |||
| Resolved | SNowick_WMF | T436277 iOS For You AB Test Data Validation |
Beta Validation in Progress for release for agent.app_version 6153
Issues with data: These have been discussed with @Tsevener - issue 1 will be resolved by persisting experiments."assigned"/experiments."enrolled"` values to users throughout the entire Home Feed flow, will test fix for #2 before Prod release.
Noting here: We integrated our data in GrowthBook using experiment_exposure events to indicate users would be part of the instrument_name = ios-home-feed experiment.
We saw an issue where Experiment Health metrics reported only logged out users in experiment. This does not indicate that we are only looking at Logged Out users in Experiment data, it just means all users looked as if they were Logged Out at initial experiment launch, Logged In users were just not detected, they are still present in the results data. We cannot use Login status as a dimension to split results on.
This will be researched by @Tsevener at a later date, the speculative cause is that users are hitting the experiment_exposure event BEFORE they are going through the authentication flow so actual. login status is not detected. This has not been verified yet but seems most likely cause.
Also note: We did a similar integration with Android data for https://phabricator.wikimedia.org/T432385 (instrument_name = new-within-interest) where we are not seeing the same issue with Logged In/Permanent users being undetected at experiment launch.