Page MenuHomePhabricator

Investigate data loss for logged-out retention rate experiment
Closed, ResolvedPublic2 Estimated Story Points

Description

On Thursday 7/31, we observed a sudden drop in event produce rate for Reader Growth's LoggedOutRetentionVisit experiment:

Screenshot 2025-07-31 at 3.28.17 PM.png (1×3 px, 335 KB)

The above cliff drop coincided with the train cut to group2 wikis at 7/31/25 UTC 19:32 - https://phabricator.wikimedia.org/T396373#11051829

After much head-scratching, we eventually figured out that the patch to update the experiment name had been backported to 1.45.0-wmf.11 but hadn't been merged to master.

The resolution was to backport to 1.45.0-wmf.12 which ended up restoring the event produce rate for that experiment:

Screenshot 2025-08-01 at 11.50.13 AM.png (1×2 px, 314 KB)

https://grafana.wikimedia.org/goto/tv2sCjwHg?orgId=1

Slack thread: https://wikimedia.slack.com/archives/C05ERLBF0E7/p1753993920317559

As a follow up to the restored event data, the data analyst for Reader Growth asked that the experiment end date be extended to 8/31. Captured as a follow up sub-task T401145: Update Reader Growth experiment LoggedOutRetentionVisit end date

Details

Other Assignee
cjming