Maniphest T393965

Automatic analysis of synthetic A/A experiment
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	mpopov
	May 12 2025, 8:43 PM

Project Tags

Referenced Files

None

Subscribers

JVanderhoop-WMF

Description

Acceptance Criteria

Synthetic A/A experiment results are explorable in https://superset.wikimedia.org/superset/dashboard/experiment-analytics/

Implementation notes

Define metric in metrics catalog based on instrumentation specification
Register experiment for automated analysis in registry
Make sure experiment is configured in a way that it will be analyzed (e.g. with respect to start & end dates, machine-readable name): T395342: Configure synthetic A/A experiment in xLab
Verify that the experiment is analyzed/re-analyzed while it is active

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Prepare for synth A/A test	repos/product-analytics/test-kitchen/experiment-analytics-configs!5	bearloga	synth-aa-experiment	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Milimetric	T392313 [Epic] SDS 2.4.11 Run a Synthetic A/A Experiment
		Resolved		mpopov	T393965 Automatic analysis of synthetic A/A experiment
		Resolved		phuedx	T393918 Instrumentation for Synthetic A/A Test (SDS 2.4.11)

Event Timeline

mpopov created this task.May 12 2025, 8:43 PM

mpopov triaged this task as Medium priority.

mpopov set the point value for this task to 2.

mpopov added a subtask: T393918: Instrumentation for Synthetic A/A Test (SDS 2.4.11).

bearloga opened https://gitlab.wikimedia.org/repos/product-analytics/experimentation-lab/experiment-analytics-configs/-/merge_requests/5

Prepare for synth A/A test

mpopov moved this task from Ready for Development to In Progress on the Test Kitchen (Experiment Platform Sprint 6) board.May 13 2025, 1:20 PM

mpopov moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

mpopov moved this task from In Progress to BLOCKED on the Test Kitchen (Experiment Platform Sprint 6) board.May 14 2025, 6:19 PM

Milimetric edited projects, added: Test Kitchen (Experiment Platform Sprint 7); removed: Test Kitchen (Experiment Platform Sprint 6).May 27 2025, 12:09 PM

Milimetric moved this task from Ready for Development to BLOCKED on the Test Kitchen (Experiment Platform Sprint 7) board.

bearloga merged https://gitlab.wikimedia.org/repos/product-analytics/experimentation-lab/experiment-analytics-configs/-/merge_requests/5

Prepare for synth A/A test

mpopov updated the task description. (Show Details)May 30 2025, 3:30 PM

Maintenance_bot removed a project: Patch-For-Review.May 30 2025, 3:31 PM

mpopov moved this task from BLOCKED to Done on the Test Kitchen (Experiment Platform Sprint 7) board.Jun 3 2025, 3:20 PM

mpopov updated the task description. (Show Details)Jun 6 2025, 1:12 PM

mpopov moved this task from Doing to Done on the Product-Analytics (Kanban) board.

Milimetric closed this task as Resolved.Jun 9 2025, 3:09 PM

Milimetric closed subtask T393918: Instrumentation for Synthetic A/A Test (SDS 2.4.11) as Resolved.