Initial Empty Search A/B test Deployment
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	bwang
	Jan 17 2025, 3:16 PM

Description

Background

This ticket is for the initial deploy of the ab test on Feb 11

Requirements

The fixes for https://phabricator.wikimedia.org/T383936 have been backported to production:
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/1118558 is backported
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/1120641 is merged
The fix for T386229: No events being logged to product_metrics.web_base.search_ab_test_session_ticks is backported
Feb 11: euwiki, cawiki
bucketing sampling rate: 99/1 (0.5, 0.5)
session length sampling rate : 50%
click tracking sampling rate: 50%

Sign off steps

Make sure the deployment steps have been documented and the analyst knows which data to ignore as part of the A/B test due to bugs.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Fix name of ABTestEnrollment configuration	operations/mediawiki-config	master	+3 -3
	Deploy web ab test to eu and ca wiki for an initial test	operations/mediawiki-config	master	+66 -8

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Jdrewniak	T379707 [Hypothesis] [3.1.8] Content Discovery: Build Feature Additional Article Recommendations (Search A/B test and Article-recommendations A/B test)
Resolved	Jdrewniak	T378094 [Epic] A/B test search suggestions on mobile
Resolved	Jdlrobson-WMF	T384019 Initial Empty Search A/B test Deployment
Resolved	bwang	T386142 Analysis of the deployment plan for the Empty Search A/B test

Event Timeline

bwang created this task.Jan 17 2025, 3:16 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 17 2025, 3:16 PM

bwang triaged this task as High priority.Jan 17 2025, 3:42 PM

bwang moved this task from Incoming to Sprint Backlog on the Web-Team board.

Jdlrobson changed the task status from Open to In Progress.Jan 17 2025, 4:56 PM

Jdlrobson moved this task from Sprint Backlog to Q3 on the Web-Team board.

Jdlrobson assigned this task to Jdrewniak.Jan 17 2025, 5:01 PM

Per team estimation

Jdrewniak removed Jdrewniak as the assignee of this task.Jan 24 2025, 4:06 PM

Jdrewniak subscribed.

Jdlrobson-WMF added a parent task: T370521: [EPIC] Test UI of recommendations in the empty state of search and compare to other locations.Feb 3 2025, 7:15 PM

@bwang to address the TBDs.

SToyofuku-WMF moved this task from Sprint Backlog to Q3 Sprint 2 (Feb 5 - Feb 19, 2025) on the Web-Team board.Feb 5 2025, 6:23 PM

SToyofuku-WMF edited projects, added Web-Team (Q3 Sprint 2 (Feb 5 - Feb 19, 2025)); removed Web-Team.

ovasileva added a project: User-notice.Feb 6 2025, 12:18 PM

For tech news: As part of this year's work around improving the ways readers discover content on the wikis, we will be running an experiment with a small number of readers that displays some suggestions for related or interesting articles within the search bar. Please check out our project page for more info.

Quiddity moved this task from To Triage to In current Tech/News draft on the User-notice board.Feb 7 2025, 10:23 PM

ovasileva edited parent tasks, added: T378094: [Epic] A/B test search suggestions on mobile; removed: T370521: [EPIC] Test UI of recommendations in the empty state of search and compare to other locations.Feb 10 2025, 9:07 AM

Jdlrobson-WMF updated the task description. (Show Details)Feb 10 2025, 5:47 PM

Deploy dates:

Test deploy: Feb 10, or feb 11
Deploy: Feb 12, or Feb 17

Jdlrobson-WMF moved this task from Ready for Development to In Progress on the Web-Team (Q3 Sprint 2 (Feb 5 - Feb 19, 2025)) board.Feb 10 2025, 10:23 PM

From refinement meeting:

We will begin with deployments to cawiki and euwiki which will help us determine whether any increases or decreases to the sampling rate will be necessary
After 2/3 days of data collection, we will continue with a deployment to all wikis

ovasileva updated the task description. (Show Details)Feb 11 2025, 2:59 PM

Change #1118566 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[operations/mediawiki-config@master] Deploy web ab test to eu and ca wiki for an initial test

https://gerrit.wikimedia.org/r/1118566

gerritbot added a project: Patch-For-Review.Feb 11 2025, 5:07 PM

Change #1118566 merged by jenkins-bot:

[operations/mediawiki-config@master] Deploy web ab test to eu and ca wiki for an initial test

https://gerrit.wikimedia.org/r/1118566

Mentioned in SAL (#wikimedia-operations) [2025-02-11T22:33:57Z] <jdrewniak@deploy2002> Started scap sync-world: Backport for [[gerrit:1118566|Deploy web ab test to eu and ca wiki for an initial test (T384019)]]

Mentioned in SAL (#wikimedia-operations) [2025-02-11T22:36:52Z] <jdrewniak@deploy2002> jdrewniak, bwang: Backport for [[gerrit:1118566|Deploy web ab test to eu and ca wiki for an initial test (T384019)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-02-11T22:48:02Z] <jdrewniak@deploy2002> Finished scap sync-world: Backport for [[gerrit:1118566|Deploy web ab test to eu and ca wiki for an initial test (T384019)]] (duration: 14m 05s)

Maintenance_bot removed a project: Patch-For-Review.Feb 11 2025, 11:30 PM

Jdlrobson-WMF updated the task description. (Show Details)Feb 12 2025, 4:24 PM

Jdlrobson-WMF updated the task description. (Show Details)

No events being logged to product_metrics_web_base_search_ab_test_clicks superset

There is an issue with the config. I will push up a fix.

Change #1119555 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] Fix name of ABTestEnrollment configuration

https://gerrit.wikimedia.org/r/1119555

gerritbot added a project: Patch-For-Review.Feb 13 2025, 7:32 PM

Change #1119555 merged by jenkins-bot:

[operations/mediawiki-config@master] Fix name of ABTestEnrollment configuration

https://gerrit.wikimedia.org/r/1119555

Mentioned in SAL (#wikimedia-operations) [2025-02-13T21:28:31Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1119555|Fix name of ABTestEnrollment configuration (T384019)]]

Maintenance_bot removed a project: Patch-For-Review.Feb 13 2025, 9:30 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-13T21:31:15Z] <tgr@deploy2002> jdlrobson, tgr: Backport for [[gerrit:1119555|Fix name of ABTestEnrollment configuration (T384019)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-02-13T21:47:55Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1119555|Fix name of ABTestEnrollment configuration (T384019)]] (duration: 19m 24s)

I managed to test this in browser and could see the events firing. There is a lag so I won't be able to confirm the events got through until approx 6pm PST

I can confirm events are now coming in for click tracking

Quiddity moved this task from In current Tech/News draft to Already announced/Archive on the User-notice board.Feb 14 2025, 8:26 PM

bwang renamed this task from Deploy Empty Search A/B test to Initial Empty Search A/B test Deployment.Feb 18 2025, 6:08 PM

bwang updated the task description. (Show Details)

I've updated this task description now that we had our initial deploy last week. I filed T386734 to account for the updates/fixes for the AB test. I think this can be signed off

@bwang can this be moved to sign off? Seems like we finished the initial deploy to cawiki and euwiki.

Suggestion: Perhaps https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/1120641 can be included in this for completeness?

Per discussion. I can sign this one off.

bwang closed subtask T386142: Analysis of the deployment plan for the Empty Search A/B test as Resolved.Feb 18 2025, 11:11 PM

bwang updated the task description. (Show Details)

@bwang while doing sign off steps I noticed we have T386734 but not deploy ticket for the actual A/B test. Can you create it (or expand the scope of T386734 to cover the actual deploy)? Or is the plan for deployment to happen in sprint 4?

We now have a deploy ticket: T386849: Deploy empty search AB test to all wikis
Deploy plan is captured in https://docs.google.com/document/d/1U5dMQJcVhpRXKuChig5770loE7099br68-tubZq2M0E/edit?tab=t.0#task=7TlshO93shIQIdQq

Jdlrobson-WMF updated the task description. (Show Details)Feb 19 2025, 5:24 PM

Maintenance_bot edited projects, added User-notice-archive; removed User-notice.Mar 1 2025, 5:32 PM

Initial Empty Search A/B test DeploymentClosed, ResolvedPublic3 Estimated Story PointsActions