Start new A/A test for retenion
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	ovasileva
	Nov 4 2025, 3:10 PM

Description

Background

We would like to continue with regular A/A tests for retention for logged out readers so we can establish seasonality of existing data

Hypothesis: If we run a secondary A/A test on retention rate for logged-out users, we will begin to establish seasonal trends for retention rate that we can use for future quarters

Requirements

Set up A/A test with same parameters as previous one - https://mpic.wikimedia.org/experiment/logged-out-retention-round2
Launch test as soon as possible - scheduled for Nov 21 & turned on

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	xLab: instrument retention baseline's page load	mediawiki/extensions/WikimediaEvents	master	+30 -6

Customize query in gerrit

Related Objects

Mentioned In: T419191: Start new A/A test for reader retention for March 2026
T414362: Remove expired instrumentation from WikimediaEvents

Event Timeline

ovasileva created this task.Nov 4 2025, 3:10 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 4 2025, 3:10 PM

ovasileva triaged this task as High priority.Nov 4 2025, 3:10 PM

ovasileva moved this task from Incoming/Inbox to Needs Refinement on the Reader Growth Team board.

ovasileva updated the task description. (Show Details)Nov 4 2025, 5:17 PM

HSwan-WMF set the point value for this task to 2.Nov 5 2025, 5:39 PM

HSwan-WMF moved this task from Needs Refinement to Ready on the Reader Growth Team board.Nov 5 2025, 5:51 PM

mfossati claimed this task.Nov 12 2025, 10:58 AM

SherryYang-WMF moved this task from Ready to Sprint 4 (Nov 12 - Nov 25) Q2 25/26) on the Reader Growth Team board.Nov 12 2025, 5:33 PM

SherryYang-WMF edited projects, added Reader Growth Team (Sprint 4 (Nov 12 - Nov 25) Q2 25/26)); removed Reader Growth Team.

mfossati changed the task status from Open to In Progress.Nov 14 2025, 5:30 PM

mfossati moved this task from Committed to Doing on the Reader Growth Team (Sprint 4 (Nov 12 - Nov 25) Q2 25/26)) board.

Change #1206429 had a related patch set uploaded (by Marco Fossati; author: Marco Fossati):

[mediawiki/extensions/WikimediaEvents@master] xLab: instrument retention baseline's page load

https://gerrit.wikimedia.org/r/1206429

gerritbot added a project: Patch-For-Review.Nov 17 2025, 6:13 PM

Elicited code review from Experiment Platform.

Change #1206429 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] xLab: instrument retention baseline's page load

https://gerrit.wikimedia.org/r/1206429

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.3; 2025-11-19).Nov 17 2025, 8:00 PM

Maintenance_bot removed a project: Patch-For-Review.Nov 17 2025, 8:32 PM

It's all set, I'll be waiting for @jwang 's final confirmation on raising the sampling rate to 10%, then we can launch.

I'll launch as soon as the related patch is deployed to target wikis.

mfossati updated the task description. (Show Details)Nov 19 2025, 3:25 PM

mfossati moved this task from QA to Signoff on the Reader Growth Team (Sprint 4 (Nov 12 - Nov 25) Q2 25/26)) board.

Verified that the related patch works on group 1 Wikipedias (Catalan, Hebrew, Italian). It's all set, I'll close this ticket as soon as the experiment starts.

Experiment started, Kafka events look fine: kafkacat -C -b kafka-jumbo1016.eqiad.wmnet:9092 -t eqiad.product_metrics.web_base -o -10 | jq 'select(.experiment.enrolled == "logged-out-retention-round2")'.
Closing.

matthiasmullie mentioned this in T414362: Remove expired instrumentation from WikimediaEvents.Jan 13 2026, 4:33 PM

KReid-WMF mentioned this in T419191: Start new A/A test for reader retention for March 2026.Mar 6 2026, 1:23 AM

Start new A/A test for retenionClosed, ResolvedPublic2 Estimated Story PointsActions