Page MenuHomePhabricator
Create Task
Maniphest T406013

Configure analytics dashboards for ImageBrowsing's A/B test
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

NOTE: This work will be blocked until we can do Data QA / "smoke test" – task TBD

The machine-readable experiment name on MPIC is fy2025-26-we3.1-image-browsing-ab-test (note that the direct link doesn't work).
See instrumentation spec and tickets: T403255: Instrument image browser's primary metric, T403259: Instrument image browsing interactions, T405489: Instrument image browser's guardrail metric.

Acceptance criteria

  • Analytics dashboards are set up
    • Defined metrics in metrics_catalog.yaml
    • Experiment registered with defined metrics in experiments_registry.yaml.

Details

Other Assignee
jwang

Related Objects
Search...

Event Timeline

mfossati created this task.Sep 30 2025, 9:49 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 30 2025, 9:49 AM
mfossati added a parent task: T398495: [Epic] Image Browsing: Set up analytics and instrumentation.Sep 30 2025, 9:50 AM
mfossati added a project: FY2025-26 WE3.1 Engaging New Audiences.
ovasileva set the point value for this task to 5.Sep 30 2025, 4:33 PM
ovasileva moved this task from Incoming/Inbox to Ready on the Reader Growth Team board.
ovasileva removed the point value 5 for this task.
ovasileva moved this task from Ready to Incoming/Inbox on the Reader Growth Team board.
ovasileva triaged this task as High priority.Sep 30 2025, 4:38 PM
ovasileva updated Other Assignee, added: jwang.
ovasileva moved this task from Incoming/Inbox to Ready on the Reader Growth Team board.
jwang claimed this task.Oct 8 2025, 3:59 PM
jwang added a project: Product-Analytics.
jwang updated the task description. (Show Details)Oct 9 2025, 10:15 PM
ovasileva moved this task from Ready to Sprint 2 (Oct 14 - Oct 27) Q2 25/26 on the Reader Growth Team board.Oct 10 2025, 3:04 PM
ovasileva edited projects, added: Reader Growth Team (Sprint 2 (Oct 14 - Oct 27) Q2 25/26); removed: Reader Growth Team.
KSarabia-WMF subscribed.Oct 14 2025, 2:21 PM
egardner updated the task description. (Show Details)Oct 14 2025, 4:59 PM
egardner set the point value for this task to 0.
mfossati mentioned this in T408104: ImageBrowsing: instrumentation production data QA.Oct 23 2025, 12:05 PM
ovasileva edited projects, added: Reader Growth Team (Sprint 3 (Oct 28 - Nov 10) Q2 25/26); removed: Reader Growth Team (Sprint 2 (Oct 14 - Oct 27) Q2 25/26).Oct 28 2025, 4:24 PM
mfossati mentioned this in T408105: ImageBrowsing: launch the A/B test on first set of wikis.Oct 28 2025, 6:03 PM
mpopov subscribed.Nov 3 2025, 6:05 PM
jwang edited projects, added: Product-Analytics (Kanban); removed: Product-Analytics.Nov 4 2025, 10:51 PM
jwang moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Nov 7 2025, 5:59 AM
egardner moved this task from Committed to Signoff on the Reader Growth Team (Sprint 3 (Oct 28 - Nov 10) Q2 25/26) board.Nov 7 2025, 10:50 PM
egardner moved this task from Signoff to Doing on the Reader Growth Team (Sprint 3 (Oct 28 - Nov 10) Q2 25/26) board.Nov 7 2025, 10:53 PM
SherryYang-WMF moved this task from Sprint 3 (Oct 28 - Nov 10) Q2 25/26 to Sprint 4 (Nov 12 - Nov 25) Q2 25/26) on the Reader Growth Team board.Nov 12 2025, 5:34 PM
SherryYang-WMF edited projects, added: Reader Growth Team (Sprint 4 (Nov 12 - Nov 25) Q2 25/26)); removed: Reader Growth Team (Sprint 3 (Oct 28 - Nov 10) Q2 25/26).
jwang added a comment.Nov 15 2025, 1:00 AM

Status update:
Have prepared the configuration code (merge request). It's under review now.

jwang added a comment.EditedNov 22 2025, 12:33 AM

Status update: Task is completed.

  • The metrics are now enabled on the analytics dashboard on xlab superset. Metrics include:
    • Mobile web image browsing clickthrough rate per user
    • 2nd week retention rate ( both 7-day cohort and full experiment cohort)
    • 2nd day retention rate (both 7-day cohort and full experiment cohort)
    • 2nd day and 3rd day retention rate (both 7-day cohort and full experiment cohort)
    • 21-day cumulative retention (both 7-day cohort and 14-day cohort)
  • The bug in CTR was fixed by merge
HSwan-WMF moved this task from Sprint 4 (Nov 12 - Nov 25) Q2 25/26) to Sprint 5 (Nov 25 - Dec 9) Q2 25/26 on the Reader Growth Team board.Nov 25 2025, 5:10 PM
HSwan-WMF edited projects, added: Reader Growth Team (Sprint 5 (Nov 25 - Dec 9) Q2 25/26); removed: Reader Growth Team (Sprint 4 (Nov 12 - Nov 25) Q2 25/26)).
jwang added a comment.Dec 1 2025, 10:04 PM

Status update:
Discovered that 2nd round AB test on enwiki changed the treatment group name from image-browsing-test to treatment. Updated the SQL so that it correctly pulls from both tiers of group results. Merge request.

Here are the dashboard links for the two AB test runs.

SherryYang-WMF moved this task from Sprint 5 (Nov 25 - Dec 9) Q2 25/26 to Sprint 6 (Dec 10 - EOY) Q2 25/26 on the Reader Growth Team board.Dec 10 2025, 4:22 PM
SherryYang-WMF edited projects, added: Reader Growth Team (Sprint 6 (Dec 10 - EOY) Q2 25/26); removed: Reader Growth Team (Sprint 5 (Nov 25 - Dec 9) Q2 25/26).
SherryYang-WMF closed this task as Resolved.Dec 16 2025, 7:24 PM
matthiasmullie added a project: ReaderExperiments-ImageBrowsing.Jan 6 2026, 1:14 PM
Restricted Application edited projects, added: Reader Growth Team; removed: Reader Growth Team (Sprint 6 (Dec 10 - EOY) Q2 25/26). · View Herald TranscriptJan 6 2026, 1:14 PM
Aklapper edited projects, added: Reader Growth Team (Sprint 6 (Dec 10 - EOY) Q2 25/26); removed: Reader Growth Team.Jan 6 2026, 2:00 PM
jwang mentioned this in T406143: Create retention metrics on xlab dashboard.Jan 8 2026, 6:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits