Page MenuHomePhabricator
Create Task
Maniphest T415611

Set up measurement plan and instrumentation spec for mobile TOC
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Background

We're planning on running an experiment to compare the new mobile TOC to the existing default behavior on the mobile site.

Hypothesis: If we test adding table of contents (TOC) to Minerva (mobile web), we will see at least one treatment show statistically significant improvement to logged-out reader retention.

Acceptance criteria

  • Set up measurement plan and instrumentation spec for mobile TOC

Metrics:

Primary:

Logged-out reader retention (21d cumulative)

Secondary:

TOC open rate (treatments only )
CTR of section topic link on TOC (treatments only )

Curiosities:

Session length
2nd week  retention rate

Design

https://phabricator.wikimedia.org/T410325

Details

Other Assignee
jwang
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Prefer assigned group reported by test kitchenmediawiki/extensions/ReaderExperimentsmaster+20 -60
Remove isInAnyTreatmentGroupmediawiki/extensions/ReaderExperimentsmaster+25 -63
Minerva TOC: Fix TOC instrumentation selectorsmediawiki/extensions/ReaderExperimentsmaster+2 -2
Minerva TOC: Fix TOC instrumentation selectorsmediawiki/extensions/ReaderExperimentswmf/1.46.0-wmf.16+2 -2
Minerva TOC Instrumentationmediawiki/extensions/ReaderExperimentsmaster+295 -45
Shared stream for reader experimentsoperations/mediawiki-configmaster+40 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

SherryYang-WMF created this task.Jan 27 2026, 12:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 27 2026, 12:06 AM
SherryYang-WMF triaged this task as High priority.Jan 27 2026, 12:06 AM
SherryYang-WMF added a project: Reader Growth Team.
SherryYang-WMF updated Other Assignee, added: jwang.
SherryYang-WMF moved this task from Incoming/Inbox to Ready on the Reader Growth Team board.
HSwan-WMF assigned this task to KSarabia-WMF.Jan 28 2026, 3:45 PM
HSwan-WMF removed KSarabia-WMF as the assignee of this task.
HSwan-WMF added a subscriber: KSarabia-WMF.
KSarabia-WMF claimed this task.Jan 28 2026, 6:40 PM
SherryYang-WMF updated the task description. (Show Details)Jan 29 2026, 11:42 PM
SherryYang-WMF updated the task description. (Show Details)
SherryYang-WMF updated the task description. (Show Details)
SherryYang-WMF mentioned this in T416051: Prod data QA for Mobile TOC.Jan 30 2026, 11:25 PM
SherryYang-WMF mentioned this in T416054: Configure analytics dashboards for Mobile TOC A/B and A/C tests.Jan 30 2026, 11:29 PM
SherryYang-WMF moved this task from Ready to Sprint 2 (Feb 3 - 16) Q3 25/26 on the Reader Growth Team board.Jan 31 2026, 12:25 AM
SherryYang-WMF edited projects, added Reader Growth Team (Sprint 2 (Feb 3 - 16) Q3 25/26); removed Reader Growth Team.
jwang updated the task description. (Show Details)Feb 4 2026, 12:49 AM
jwang subscribed.Feb 4 2026, 12:52 AM

Status update:
Here are the Mobile Web ToC Measurement Plan and Instrumentation Spec. Ready for product team review.

matthiasmullie merged a task: T413405: [Minerva TOC] Instrumentation placeholder.Feb 4 2026, 5:43 PM
matthiasmullie subscribed.
egardner set the point value for this task to 3.Feb 4 2026, 6:05 PM
egardner subscribed.

Note – this task will also encompass the work to add the actual instrumentation code to our feature. Setting up dashboards can be handled in a separate task.

mfossati subscribed.Feb 5 2026, 10:09 PM

This patch removed expired instrumentation, but I think it's gonna be useful for this task, since we've already used page-visit and session tick events in past experiments.

KSarabia-WMF moved this task from Committed to Doing on the Reader Growth Team (Sprint 2 (Feb 3 - 16) Q3 25/26) board.Mon, Feb 9, 5:03 PM
gerritbot added a comment.Tue, Feb 10, 8:05 PM

Change #1238422 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/ReaderExperiments@master] [WIP] Minerva TOC Instrumentation

https://gerrit.wikimedia.org/r/1238422

gerritbot added a project: Patch-For-Review.Tue, Feb 10, 8:06 PM
lwatson added a parent task: T415948: [EPIC] Minerva TOC.Wed, Feb 11, 11:49 PM
lwatson added a project: ReaderExperiments-MinervaTOC.Wed, Feb 11, 11:52 PM
KSarabia-WMF added a comment.Thu, Feb 12, 8:18 PM

QA Instructions

  1. Use Minerva and main namespace pages for all checks. Example: ?useskin=minerva on a mainspace article.
  2. Verify enrollment overrides work:
    • URL override: ?mpo=fy25-26-we3.10.2-mobile-toc-abc-test:treatment1 and :treatment2.
    • Control override: use the actual control group name from TestKitchen, which should be 'control'
  1. For any assigned group (control + treatments), confirm Parsoid is forced.
  1. Control group behavior:
    • Confirm instrumentation module loads. In console: mw.loader.getState('ext.readerExperiments.minervaToc.instrumentation') should be ready.
    • Confirm sticky/button UI does not appear unless treatments are selected.
    • Confirm sections remain collapsible (no behavior change).
  1. Instrumentation events:
    • Add ?tocInstrumentationDebug=1 and open console for [MinervaToc][QA] logs.
    • On page load, verify page-visited logs as sent or queued then flushed.
    • Open TOC and verify init_toc log.
      • Click the TOC icon to open; verify click with action_context: 'toc_icon'
      • Click a link inside TOC contents; verify click with action_context: 'topic_link' and action_source: 'toc'.
mfossati moved this task from Doing to Code Review on the Reader Growth Team (Sprint 2 (Feb 3 - 16) Q3 25/26) board.Tue, Feb 17, 2:32 PM
SherryYang-WMF moved this task from Sprint 2 (Feb 3 - 16) Q3 25/26 to Sprint 3 (Feb 17 - Mar 2) Q3 25/26 on the Reader Growth Team board.Tue, Feb 17, 5:42 PM
SherryYang-WMF edited projects, added Reader Growth Team (Sprint 3 (Feb 17 - Mar 2) Q3 25/26); removed Reader Growth Team (Sprint 2 (Feb 3 - 16) Q3 25/26).
egardner moved this task from Committed to Code Review on the Reader Growth Team (Sprint 3 (Feb 17 - Mar 2) Q3 25/26) board.Tue, Feb 17, 6:33 PM
gerritbot added a comment.Wed, Feb 18, 12:33 PM

Change #1240264 had a related patch set uploaded (by Marco Fossati; author: Marco Fossati):

[operations/mediawiki-config@master] ReaderExperiments' MobileToc stream configuration

https://gerrit.wikimedia.org/r/1240264

mfossati moved this task from Code Review to Doing on the Reader Growth Team (Sprint 3 (Feb 17 - Mar 2) Q3 25/26) board.Wed, Feb 18, 12:38 PM

Events review

Instrumentation spec: https://docs.google.com/spreadsheets/d/11zMRr-CDJzpdzxhunk8k7hw7rNfg6D8IZazVLZHpP5g/edit?gid=0#gid=0

  • page load: action: page-visited ✅
  • ToC icon click: action: click action_context: toc_icon ✅
  • ToC section link click: action: click action_context: topic_link action_source=toc ✅
  • ToC shows: action: init_toc - I don't think this is needed, seems duplicate of ToC icon click
  • session length: action: tick action_context: "N" ✅
gerritbot added a comment.Thu, Feb 19, 12:14 PM

Change #1238422 merged by jenkins-bot:

[mediawiki/extensions/ReaderExperiments@master] Minerva TOC Instrumentation

https://gerrit.wikimedia.org/r/1238422

gerritbot added a comment.Thu, Feb 19, 12:15 PM

Change #1240659 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[mediawiki/extensions/ReaderExperiments@master] Remove isInAnyTreatmentGroup

https://gerrit.wikimedia.org/r/1240659

gerritbot added a comment.Thu, Feb 19, 12:15 PM

Change #1240660 had a related patch set uploaded (by Matthias Mullie; author: Matthias Mullie):

[mediawiki/extensions/ReaderExperiments@master] Prefer assigned group reported by test kitchen

https://gerrit.wikimedia.org/r/1240660

gerritbot added a comment.Thu, Feb 19, 12:29 PM

Change #1240264 merged by jenkins-bot:

[operations/mediawiki-config@master] Shared stream for reader experiments

https://gerrit.wikimedia.org/r/1240264

Stashbot added a comment.Thu, Feb 19, 12:36 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-19T12:36:42Z] <mlitn@deploy2002> Started scap sync-world: Backport for [[gerrit:1240264|Shared stream for reader experiments (T415611)]]

Stashbot added a comment.Thu, Feb 19, 12:38 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-19T12:38:53Z] <mlitn@deploy2002> mfossati, mlitn: Backport for [[gerrit:1240264|Shared stream for reader experiments (T415611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Thu, Feb 19, 12:44 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-19T12:44:24Z] <mlitn@deploy2002> Finished scap sync-world: Backport for [[gerrit:1240264|Shared stream for reader experiments (T415611)]] (duration: 07m 42s)

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.17; 2026-02-24).Thu, Feb 19, 1:00 PM
mfossati moved this task from Doing to QA on the Reader Growth Team (Sprint 3 (Feb 17 - Mar 2) Q3 25/26) board.Thu, Feb 19, 2:46 PM
gerritbot added a comment.Thu, Feb 19, 9:53 PM

Change #1240809 had a related patch set uploaded (by Eric Gardner; author: Eric Gardner):

[mediawiki/extensions/ReaderExperiments@master] Minerva TOC: Fix TOC instrumentation selectors

https://gerrit.wikimedia.org/r/1240809

gerritbot added a comment.Thu, Feb 19, 10:18 PM

Change #1240814 had a related patch set uploaded (by Eric Gardner; author: Eric Gardner):

[mediawiki/extensions/ReaderExperiments@wmf/1.46.0-wmf.16] Minerva TOC: Fix TOC instrumentation selectors

https://gerrit.wikimedia.org/r/1240814

gerritbot added a comment.Thu, Feb 19, 10:22 PM

Change #1240809 merged by jenkins-bot:

[mediawiki/extensions/ReaderExperiments@master] Minerva TOC: Fix TOC instrumentation selectors

https://gerrit.wikimedia.org/r/1240809

gerritbot added a comment.Thu, Feb 19, 10:24 PM

Change #1240814 merged by jenkins-bot:

[mediawiki/extensions/ReaderExperiments@wmf/1.46.0-wmf.16] Minerva TOC: Fix TOC instrumentation selectors

https://gerrit.wikimedia.org/r/1240814

Stashbot added a comment.Thu, Feb 19, 10:24 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-19T22:24:31Z] <egardner@deploy2002> Started scap sync-world: Backport for [[gerrit:1240814|Minerva TOC: Fix TOC instrumentation selectors (T415611)]]

Stashbot added a comment.Thu, Feb 19, 10:26 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-19T22:26:26Z] <egardner@deploy2002> egardner: Backport for [[gerrit:1240814|Minerva TOC: Fix TOC instrumentation selectors (T415611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Thu, Feb 19, 10:32 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-19T22:32:03Z] <egardner@deploy2002> Finished scap sync-world: Backport for [[gerrit:1240814|Minerva TOC: Fix TOC instrumentation selectors (T415611)]] (duration: 07m 31s)

ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.16; 2026-02-17); removed MW-1.46-notes (1.46.0-wmf.17; 2026-02-24).Thu, Feb 19, 11:00 PM
gerritbot added a comment.Fri, Feb 20, 7:47 PM

Change #1240659 merged by jenkins-bot:

[mediawiki/extensions/ReaderExperiments@master] Remove isInAnyTreatmentGroup

https://gerrit.wikimedia.org/r/1240659

gerritbot added a comment.Fri, Feb 20, 7:47 PM

Change #1240660 merged by jenkins-bot:

[mediawiki/extensions/ReaderExperiments@master] Prefer assigned group reported by test kitchen

https://gerrit.wikimedia.org/r/1240660

egardner closed this task as Resolved.Fri, Feb 20, 7:50 PM
ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.17; 2026-02-24); removed MW-1.46-notes (1.46.0-wmf.16; 2026-02-17).Fri, Feb 20, 8:00 PM
Maintenance_bot removed a project: Patch-For-Review.Fri, Feb 20, 8:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits