Page MenuHomePhabricator

Mobile Account Menu Experiment: Investigate Sample Ratio Mismatch (Hypothesis: WE1.8.4)
Closed, ResolvedPublic

Description

Investigate a potential Sample Ratio Mismatch in the Mobile Account Menu experiment. Validate whether traffic allocation aligns with the expected experiment split and identify any instrumentation, logging, or platform-related issues that could explain the discrepancy.

Background

Initial analysis suggests a Sample Ratio Mismatch (SRM) between the control and treatment.

Current experiment data when filtered to "WE 1.8 FY25-26 Mobile Account Menu" and "Logged-out only":

Screenshot 2026-04-12 at 6.14.23 AM.png (736×246 px, 49 KB)

This may indicate issues with experiment setup, event logging, or downstream data processing. Resolving SRM is required before metrics can be considered reliable.

Acceptance Criteria
  • Review instrumentation, experiment configuration, and intended traffic split
  • Growth engineering collaborates with Product Analytics and Experiment Platform teams as needed to validate experiment setup and assignment logic
  • Root cause of Sample Ratio Mismatch is identified and documented
  • Any issues with instrumentation, experiment setup, or data pipelines are resolved or have clear follow-up tasks
  • Experiment is paused/reset if needed

Event Timeline

KStoller-WMF renamed this task from WE 1.8 FY25-26 Mobile Account Menu: Investigate Sample Ratio Mismatch to Mobile Account Menu Experiment: Investigate Sample Ratio Mismatch (Hypothesis: WE1.8.4).
KStoller-WMF triaged this task as High priority.
KStoller-WMF moved this task from Inbox to Needs Refinement on the Growth-Team board.
KStoller-WMF claimed this task.

The experiment split remains slightly uneven but still very close to the expected allocation:

  • Control: 979822 subjects (49.9%)
  • Treatment: 985371 subjects (50.1%)

After reviewing this with Product Analytics and the Experimentation Platform team, the SRM warning is likely a false positive. We believe the assignment is functioning as expected, and the samples can be treated as effectively random. As a result, the experiment results should be considered valid.

At this scale, SRM checks become extremely sensitive. With millions of users, even a negligible deviation from a perfect 50/50 split (for example, 50.01% vs. 49.99%) can produce a statistically significant chi-square result (p < 0.001), despite having no practical impact on randomization quality.

This highlights an important principle: large sample sizes increase statistical power to the point where even trivial differences are detectable. In this case, the SRM signal reflects statistical sensitivity rather than a meaningful issue, reinforcing the need to consider practical significance alongside statistical significance.

(I was going to decline this task, but I think I'll just mark it as Resolved since effort was spent investigating on the Growth side and on the Experimentation Platform side. Thanks everyone who helped!)