Page MenuHomePhabricator

[EPIC] QuickSurveys improvements
Open, MediumPublic

Description

With our current QuickSurveys infrastructure, the Research team can’t reliably answer the question that product teams are asking: what actually drives reader retention?

Today, survey signals overrepresent deeply-engaged users. We know little about folks who show up once and don’t come back, which is exactly the population we’re trying to better understand. We also suffer from 20-40% event loss that is likely biased, not random. Even if we are hearing from a small proportion of relatively casual users, we may entirely miss their responses.

Why this matters now

Reader teams are focused on reader retention, and surveys are one of the few ways to connect why people come to Wikipedia to whether or not they come back. Observational data can’t do this on its own, and self-reported retention seems to be unreliable, skewed towards daily use.

What’s not working

  • Event loss is biased, not random. People lost to event loss are likely systematically different (ad blockers, no-script softwares), which distorts who we hear from.
  • Weak observational data for respondents. Webrequest data is a ~24h snapshot with rapidly degrading identifier quality, limiting our ability to compare who was sampled vs. who responded, making it hard to adjust for bias.
  • Session-based enrollment. QuickSurveys enrolls on sessions, not on individual identifiers. This is workable for large groups over short time windows (e.g. 36 hours) but breaks down for smaller segments over longer periods, in which cases we often re-survey the same finite population and have to filter out repeat responses. Additionally, session-based enrollment contributes to bias: the more sessions a user has, the more likely they are to be surveyed.

[[[ https://docs.google.com/document/d/1hYZHH6h_PHKoUYp-E1rU7PKgf51MSKJTurojJzmpIcA/edit?tab=t.vi1f2owuhbms | more context and notes here ]]]