Page MenuHomePhabricator

Disable the A/A test in production
Closed, ResolvedPublic0 Estimated Story Points

Description

Once T399486 is complete, we expect to run the A/A test in production for about one month to collect traffic data.

After a sufficient amount of time has passed, we should disable the A/A test to end the experiment. Afterwards we will have a snapshot of data that we can hopefully use to establish a good baseline for user retention, which we will then use to inform future product decisions. Data analysis work is outside the scope of this epic.

Open Questions
  • Are we sure that 1 month (30 days) is the correct amount of time here? Will we determine how long to run the test while the test is running, based on incoming data?
  • Do we need to revert T399482's instrument patch?
  • Do we need to remove the instrument from the instrument list?

See https://wikitech.wikimedia.org/wiki/Experimentation_Lab/Decommission_an_instrument.

Acceptance Criteria
  • A config patch has been merged to disable the test from all participating wikis in production

Event Timeline

ovasileva set the point value for this task to 0.Aug 27 2025, 5:12 PM

This is done. Let's discuss next test date cycle before closing this.

KSarabia-WMF moved this task from Committed to QA on the Reader Growth Team (Sprint 6) board.

@jwang Can you confirm here that retention data collection has stopped? Thank you

@egardner Can you verify the screenshots below and sign off

Before Test End Date

Screenshot 2025-09-23 at 10.40.45 AM.png (1×2 px, 277 KB)

After Test End Date

Screenshot 2025-09-23 at 10.38.29 AM.png (838×2 px, 140 KB)

@KSarabia-WMF I can log into Superset (here) and see that there is no data for this experiment.

Screenshot 2025-09-25 at 9.30.35 AM.png (2×2 px, 999 KB)
.

This is what we want, correct?