Page MenuHomePhabricator

[Truncation] Launch experiment
Closed, ResolvedPublic

Description

Overview

This task tracks the pre- and post-launch work for the Truncation / Minerva Compact Lead experiment (T432075: [Epic] Truncation experiment). It is adapted from the team's Experiment Launch Checklist.

Planned launch date: Thursday, September 17, 2026 4:00 PM (CDT)

Completed setup and implementation

  • Feature code completion
  • Core experiment implementation is complete: T432188: Instrumentation for truncation experiment
    • Experiment eligibility is limited to:
      • Anonymous/logged out users and temporary accounts
      • Main-namespace pages (Namespace 0)
      • Minerva skin (mobile view)
    • Experiment exposure events are implemented via experiment.sendExposure()
    • Experiment assignment is read from GrowthBook experiment platform data
  • Experiment is configured in GrowthBook: https://growthbook.wikimedia.org/experiment/exp_2lq00w7nmtvpm33f#overview
    • GrowthBook variations include the intended experiment experiences
    • Primary metric (21-day reader retention) configured in GrowthBook
    • Secondary and guardrail metrics configured in GrowthBook (eg. lead section expansion rate, quick facts toggle rate, edit attempts)
  • Local and development overrides are available
    • Feature is isolated behind configuration flags: $wgMinervaTruncateLeadSection and $wgMFQuickFacts
    • URL parameters:
      • ?mpo=minerva-compact-lead:control
      • ?mpo=minerva-compact-lead:trunc-lead-infobox
      • ?mpo=minerva-compact-lead:trunc-lead

Completed bug fixes & refinements

Before launch

Instrumentation and measurement

  • Confirm all events in the instrumentation specification (T432188: Instrumentation for truncation experiment) are marked complete
  • Manually verify all specified events and payloads:
    • experiment_exposure (control and treatments)
    • page_visit (control and treatments)
    • session length tick (control and treatments)
    • lead_section click interaction (expand)
    • quick_facts section click/toggle interaction (expand/collapse)
    • edit_attempt_init
  • Verify instrumentation eligibility checks for anonymous users, main-namespace pages, and Minerva skin
  • Confirm GrowthBook experiment key, variation keys, metrics, and measurement specification agree
  • Ensure that ReaderExperiments is enabled on all target wikis. T438009: Enable ReaderExperiments on all target wikis

QA and bug bash

Patched bugs awaiting final QA/signoff

  • ❌ [DECLINED] T433271: Truncation: fix "read more" button's accessibility issues ➡ Decision: non-launch blocking; team will not address this bug now; pulled the accessibility bug out of the parent bug bash and it's now a standalone ticket.
  • ❌ [DECLINED] Team decided we'd address the special characters side box issue after the experiment (More details about div.side-box metadata side-box-right contains-special-characters noprint selfref can be found in this comment: T432752#12295253).

Release readiness

  • Confirm all required patches (skins/MinervaNeue, extensions/MobileFrontend, extensions/ReaderExperiments) are deployed to production (MW 1.47.0-wmf.20 train)
  • Confirm there are no launch-blocking regressions from the final QA pass
  • Communicate the planned launch date to target wikis (Movement Communications/Eliza)
  • Confirm translation are synced/deployed for all target wikis
    • minimal-minerva-lead-section-read-more | "Read more" | translatewiki
    • mobile-frontend-quick-facts | "Quick facts" | translatewiki

Immediately before launch

  • Obtain final go/no-go signoff
  • Complete final instrumentation QA: T436199#12334621
  • Schedule or start the experiment in GrowthBook before 14:30 UTC, Monday through Thursday
  • Record the experiment start time and expected end date in this task
    • Start date/time: Thursday, September 17, 2026 4:00 PM (CDT)
    • End date: October 15, 2026

Post-launch

  • Confirm both control and treatment variation users are being enrolled across all target wikis
  • Validate experiment_exposure events in production
  • Validate page_visit, read_more button clicks, and quick_facts expand/collapse events in production
  • Validate edit_attempt_init
  • Check event volume by target wiki for unexpected gaps or sample ratio mismatches (SRM)
  • Confirm GrowthBook is receiving data for primary, secondary, and guardrail metrics
  • Document production-data QA findings in this task
  • Monitor for regressions or community feedback during the initial launch period

After the experiment

  • Stop the experiment at the agreed end date
  • Complete experiment analysis
  • Remove Truncation experiment instrumentation and temporary experiment code T437467: Remove truncation experiment

Event Timeline

etz subscribed.
lwatson changed the task status from Open to In Progress.Sep 10 2026, 12:52 PM
lwatson updated the task description. (Show Details)

Change #1339741 had a related patch set uploaded (by LWatson; author: LWatson):

[operations/mediawiki-config@master] Enable ReaderExperiments in eswiki, jawiki, and ptwiki

https://gerrit.wikimedia.org/r/1339741

Confirm GrowthBook experiment key, variation keys, metrics, and measurement specification agree

[BLOCKED] I'm unable to verify the experiment's configuration in GrowthBook. This step should be completed before launch.

More details about the GrowthBook issue:

  1. When I navigate to the experiment's URL, there's a persistent error.
    GrowthBook_error_experiment_URL.png (2,018×298 px, 53 KB)
  2. There are no existing experiments in the experiment dashboard.
    GrowthBook_error_no_experiments.png (2,012×1,076 px, 169 KB)
  3. My GrowthBook permissions are approved: readonly and custom elevated access

cc: @SherryYang-WMF

Final instrumentation QA

Confirmed experiment events and instrument loaded on target wikis using enrollment overrides(?mpo URL query params).

Target wikiURLVerified
Arabic (ar)arwiki example✅
Spanish (es)eswiki example✅
French (fr)frwiki example✅
Indonesian (id)idwiki example 1, idwiki example 2✅
Italian (it)itwiki example✅
Japanese (ja)jawiki example 1, jawiki example 2✅
Portuguese (pt)ptwiki example 1, ptwiki example 2✅
Vietnamese (vi)viwiki example✅
Traditional Chinese (zh-hant)zhwiki example✅

Notes from standup on September 17th:

  • Team reviewed the checklist to confirm readiness for launch.
  • Obtain final go/no-go signoff: Confirmed with engineers, QA, EM, PM, and design during standup on Sept. 17th.
  • Record the experiment start time and expected end date in this task:
    • Start date/time: Thursday, September 17, 2026 4:00 PM (CDT)
    • End date: October 15, 2026.
    • Experiments are manually turned off so Kim suggested that we add the end date to the team calendar.