Launch content separation (Zebra #9) A/B test
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	ovasileva
	May 4 2023, 3:16 PM

Description

Background

We are now ready to launch the a/b test for the content separation work

Acceptance criteria

Enable a/b test for the following wikis on ~~Wed May 17th~~ Monday 5/29: hewiki
Enable a/b test for the following wikis on ~~Thur May 18th~~ Tuesday 5/30: enwiki, fawiki, frwiki, kowiki, ptwiki, srwiki, thwiki, trwiki, viwiki

Signoff criteria

Ping @jwang after deployment for A/B test schema QA
Ensure A/B test is disabled two weeks after being initially enabled. The interface should go back to the control state (Edit: See follow up ticket: https://phabricator.wikimedia.org/T337956)
Document on https://www.mediawiki.org/wiki/Reading/Web/Release_timeline

Details

Subject	Repo	Branch	Lines +/-
Launch content separation Zebra AB Test	operations/mediawiki-config	master	+36 -0
Enable Vector "Zebra" AB test on Hebrew wikipedia	operations/mediawiki-config	master	+9 -2
Use document feature classes to extract A/B test state	mediawiki/skins/Vector	wmf/1.41.0-wmf.10	+5 -0
Use document feature classes to extract A/B test state	mediawiki/skins/Vector	master	+5 -0
Override mw.experiments.getBucket() via body classes	mediawiki/skins/Vector	master	+15 -1
Enable zebra ab test in hewiki	operations/mediawiki-config	master	+13 -2
Make ABRequirement aware of Require	mediawiki/skins/Vector	master	+17 -4
[Beta] Enable Vector AB test on beta spanish wiki	operations/mediawiki-config	master	+19 -0
Enable Vector "Zebra" AB test on beta cluster.	operations/mediawiki-config	master	+11 -8

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	ovasileva	T333180 [EPIC] A/B test Zebra #9 prototype
Resolved	jwang	T335972 Launch content separation (Zebra #9) A/B test
Resolved	jwang	T336299 QA A/B test schema for Zebra #9 A/B test
Resolved	bwang	T336969 [Zebra AB test] Fix the mixing of global and user IDs for AB Test Enrollment Bucketing
Resolved	KSarabia-WMF	T337956 Turn off Zebra A/B Test
Resolved	Mabualruz	T339957 Zebra: Completely remove any A/B test-related code

Event Timeline

ovasileva created this task.May 4 2023, 3:16 PM

Restricted Application added subscribers: Huji, revi, Aklapper. · View Herald TranscriptMay 4 2023, 3:16 PM

ovasileva triaged this task as High priority.May 4 2023, 3:16 PM

ovasileva added a project: Desktop Improvements (Vector 2022).

ovasileva added a parent task: T333180: [EPIC] A/B test Zebra #9 prototype.May 4 2023, 5:24 PM

LGoto set the point value for this task to 2.May 4 2023, 5:24 PM

Ladsgroup subscribed.May 4 2023, 5:32 PM

Izno subscribed.May 4 2023, 8:16 PM

ovasileva moved this task from Incoming to Ready for Development on the Web-Team-Backlog (Web Team FY2022-23 Q4 Sprint 3) board.May 8 2023, 2:21 PM

KSarabia-WMF claimed this task.May 8 2023, 4:58 PM

Jdlrobson updated the task description. (Show Details)May 8 2023, 5:24 PM

KSarabia-WMF moved this task from Ready for Development to Doing on the Web-Team-Backlog (Web Team FY2022-23 Q4 Sprint 3) board.May 9 2023, 3:40 PM

ovasileva updated the task description. (Show Details)May 9 2023, 7:06 PM

Change 918568 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[operations/mediawiki-config@master] Launch content separation Zebra AB Test

https://gerrit.wikimedia.org/r/918568

gerritbot added a project: Patch-For-Review.May 10 2023, 4:56 PM

Change 918584 had a related patch set uploaded (by Jdrewniak; author: Jdrewniak):

[operations/mediawiki-config@master] Enable Vector "Zebra" AB test on beta cluster.

https://gerrit.wikimedia.org/r/918584

Change 918584 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Vector "Zebra" AB test on beta cluster.

https://gerrit.wikimedia.org/r/918584

Jdlrobson moved this task from Doing to Blocked on Others on the Web-Team-Backlog (Web Team FY2022-23 Q4 Sprint 3) board.May 10 2023, 8:36 PM

Jdlrobson updated the task description. (Show Details)

Change 918613 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/skins/Vector@master] Make ABRequirement aware of Require

https://gerrit.wikimedia.org/r/918613

Jdlrobson moved this task from Blocked on Others to Doing on the Web-Team-Backlog (Web Team FY2022-23 Q4 Sprint 3) board.May 16 2023, 4:44 PM

Change 920386 had a related patch set uploaded (by Jdlrobson; author: Kimberly Sarabia):

[operations/mediawiki-config@master] Enable zebra ab test in hewiki

https://gerrit.wikimedia.org/r/920386

Jdlrobson moved this task from Doing to Code Review on the Web-Team-Backlog (Web Team FY2022-23 Q4 Sprint 3) board.May 16 2023, 10:44 PM

Environment	Event	Status
Beta-Cluster	`web_ab_test_enrollment`	Verified
test.wikipedia.org	`mediawiki.web_ui_scroll`	Verified
test.wikipedias.org	`editattemptstep`	Verified

I checked the patches, and we're good to deploy as follows:
Wed: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/920386
Thur: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/918568

Change 920386 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable zebra ab test in hewiki

https://gerrit.wikimedia.org/r/920386

Mentioned in SAL (#wikimedia-operations) [2023-05-17T20:17:29Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:920784|GrowthExperiments: amend wrong wiki prefix for jbowiki (T308134)]], [[gerrit:920732|NewTopicOptOutActiveUsers: Skip bot users etc. (T317375)]], [[gerrit:920386|Enable zebra ab test in hewiki (T335972)]]

Mentioned in SAL (#wikimedia-operations) [2023-05-17T20:19:04Z] <urbanecm@deploy1002> urbanecm and matmarex and ksarabia and sgimeno: Backport for [[gerrit:920784|GrowthExperiments: amend wrong wiki prefix for jbowiki (T308134)]], [[gerrit:920732|NewTopicOptOutActiveUsers: Skip bot users etc. (T317375)]], [[gerrit:920386|Enable zebra ab test in hewiki (T335972)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.

Mentioned in SAL (#wikimedia-operations) [2023-05-17T20:29:06Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:920784|GrowthExperiments: amend wrong wiki prefix for jbowiki (T308134)]], [[gerrit:920732|NewTopicOptOutActiveUsers: Skip bot users etc. (T317375)]], [[gerrit:920386|Enable zebra ab test in hewiki (T335972)]] (duration: 11m 36s)

Change 920793 had a related patch set uploaded (by Jdrewniak; author: Jdrewniak):

[operations/mediawiki-config@master] Enable Veector AB test on beta spanish wiki

https://gerrit.wikimedia.org/r/920793

Change 920793 merged by jenkins-bot:

[operations/mediawiki-config@master] [Beta] Enable Vector AB test on beta spanish wiki

https://gerrit.wikimedia.org/r/920793

LGoto edited projects, added Web-Team-Backlog (Web Team FY2022-23 Q4 Sprint 4); removed Web-Team-Backlog (Web Team FY2022-23 Q4 Sprint 3).May 22 2023, 5:20 PM

LGoto moved this task from Incoming to Blocked on Others on the Web-Team-Backlog (Web Team FY2022-23 Q4 Sprint 4) board.

Change 922564 had a related patch set uploaded (by Jdrewniak; author: Jdrewniak):

[operations/mediawiki-config@master] Enable Vector "Zebra" AB test on Hebrew wikipedia

https://gerrit.wikimedia.org/r/922564

Change 922939 had a related patch set uploaded (by Jdrewniak; author: Jdrewniak):

[mediawiki/skins/Vector@master] Override mw.experiments.getBucket() via body classes

https://gerrit.wikimedia.org/r/922939

Change 922939 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Override mw.experiments.getBucket() via body classes

https://gerrit.wikimedia.org/r/922939

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.11; 2023-05-30).May 25 2023, 1:00 AM

Change 923409 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] Use document feature classes to extract A/B test state

https://gerrit.wikimedia.org/r/923409

Change 923281 had a related patch set uploaded (by Jdrewniak; author: Jdlrobson):

[mediawiki/skins/Vector@wmf/1.41.0-wmf.10] Use document feature classes to extract A/B test state

https://gerrit.wikimedia.org/r/923281

Change 923409 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Use document feature classes to extract A/B test state

https://gerrit.wikimedia.org/r/923409

@KSarabia-WMF will backport https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/923409 to 1.41.0-wmf.10 today (Thursday 25th May) and we will aim to roll this out again early next week.

Change 923281 merged by jenkins-bot:

[mediawiki/skins/Vector@wmf/1.41.0-wmf.10] Use document feature classes to extract A/B test state

https://gerrit.wikimedia.org/r/923281

Mentioned in SAL (#wikimedia-operations) [2023-05-25T20:21:06Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:923281|Use document feature classes to extract A/B test state (T335972)]]

Mentioned in SAL (#wikimedia-operations) [2023-05-25T20:22:36Z] <samtar@deploy1002> jdrewniak and samtar: Backport for [[gerrit:923281|Use document feature classes to extract A/B test state (T335972)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-05-25T20:32:04Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:923281|Use document feature classes to extract A/B test state (T335972)]] (duration: 10m 58s)

ReleaseTaggerBot edited projects, added MW-1.41-notes (1.41.0-wmf.10; 2023-05-23); removed MW-1.41-notes (1.41.0-wmf.11; 2023-05-30).May 25 2023, 9:00 PM

FYI - This is the revised Zebra A/B Test rollout timeline:

Enable a/b test for the following wikis on ~~Wed May 17th~~ Monday 5/29: hewiki

Enable a/b test for the following wikis on ~~Thur May 18th~~ Tuesday 5/30: enwiki, fawiki, frwiki, kowiki, ptwiki, srwiki, thwiki, trwiki, viwiki

cc: @jwang @sgrabarczuk @NHillard-WMF

KSarabia-WMF updated the task description. (Show Details)May 25 2023, 11:11 PM

Update: 5/29 is a US holiday and there are no backports that day!
The following is hopefully the final revised timeline.

Enable a/b test for the following wikis on Tuesday 5/30: hewiki
Enable a/b test for the following wikis on Wednesday 5/31: enwiki, fawiki, frwiki, kowiki, ptwiki, srwiki, thwiki, trwiki, viwiki

cc: @jwang @sgrabarczuk @NHillard-WMF

jwang mentioned this in T333222: Create instrumentation spec for Zebra #9 prototype.May 30 2023, 9:54 PM

Jdrewniak mentioned this in T327718: Increase space between groups in main menu and page tools menu.Jun 1 2023, 4:19 PM

Jdrewniak moved this task from Code Review to Ready for Signoff on the Web-Team-Backlog (Web Team FY2022-23 Q4 Sprint 4) board.Jun 1 2023, 4:41 PM

Turn off test and put that ticket in Sign Off by 6/16

KSarabia-WMF updated the task description. (Show Details)Jun 1 2023, 4:47 PM

@KSarabia-WMF, Is it possible to prolong the AB test by an additional 3 or 4 days? This way, we would obtain a full two weeks of data once the edit button issue is resolved.

In T335972#8897405, @jwang wrote:

@KSarabia-WMF, Is it possible to prolong the AB test by an additional 3 or 4 days? This way, we would obtain a full two weeks of data once the edit button issue is resolved.

Yes, i'll move the deadline

KSarabia-WMF updated the task description. (Show Details)Jun 2 2023, 10:50 PM

KSarabia-WMF updated the task description. (Show Details)Jun 2 2023, 10:55 PM

Alpyne awarded a token.Jun 3 2023, 1:01 PM

LGoto reassigned this task from KSarabia-WMF to jwang.Jun 5 2023, 5:25 PM

LGoto added a subscriber: KSarabia-WMF.

The ab test events are available in mediawiki_web_ab_test_enrollment with experiment name skin-vector-zebra-experiment now.