Page MenuHomePhabricator

Analyze results of the KaiOS app homepage content suggestions experiment
Closed, ResolvedPublic

Description

Based on my power analysis (T271316) and post-deployment data check (T280830), we now have enough data to perform the final analysis of the experiment.

Event Timeline

nshahquinn-wmf created this task.

The initial analysis is done:

  • Users in the control group had an average session length of 2 min, 15 s. In the trending article group, this increased by 15% (t-test p ≪ 0.001).
  • Users in the control group had an average of 1.7 sessions. In the experiment group, this increased by 2% (Mann-Whitney U test p = 0.003)

I've been working on fitting more sophisticated hierarchical models as well. That isn't fully done, but I expect to pick it up next quarter (T285777).

My full analysis notebook is at on GitHub in wikimedia-research/2021-KaiOS-app-homepage-content-suggestions.