Page MenuHomePhabricator

iOS For You AB Test Data Validation
Closed, ResolvedPublic

Event Timeline

SNowick_WMF triaged this task as High priority.
SNowick_WMF added a project: User-SNowick_WMF.
SNowick_WMF removed subscribers: hubaishan, alaa.

Beta Validation in Progress for release for agent.app_version 6153

Issues with data: These have been discussed with @Tsevener - issue 1 will be resolved by persisting experiments."assigned"/experiments."enrolled"` values to users throughout the entire Home Feed flow, will test fix for #2 before Prod release.

  1. For the test experiments."assigned"/experiments."enrolled" values - I am seeing 77 (out of 399) unique app_install_ids that had NULL experiments."assigned"/experiments."enrolled" values - see deck. It looks like the non-assigned users were just showing up in data twice with once with groups/enrolled and again with no groups/enrolled.
  1. For all of the action: click menu events on the Cards there is no value for action_subtype - they should all have value action_subtype: feed_overflow -  see deck

Noting here: We integrated our data in GrowthBook using experiment_exposure events to indicate users would be part of the instrument_name = ios-home-feed experiment.

We saw an issue where Experiment Health metrics reported only logged out users in experiment. This does not indicate that we are only looking at Logged Out users in Experiment data, it just means all users looked as if they were Logged Out at initial experiment launch, Logged In users were just not detected, they are still present in the results data. We cannot use Login status as a dimension to split results on.

This will be researched by @Tsevener at a later date, the speculative cause is that users are hitting the experiment_exposure event BEFORE they are going through the authentication flow so actual. login status is not detected. This has not been verified yet but seems most likely cause.

Also note: We did a similar integration with Android data for https://phabricator.wikimedia.org/T432385 (instrument_name = new-within-interest) where we are not seeing the same issue with Logged In/Permanent users being undetected at experiment launch.

Prod data validation has passed for release agent.app_version 6160
Data

SNowick_WMF updated the task description. (Show Details)