Page MenuHomePhabricator

Initial Empty Search A/B test Deployment
Closed, ResolvedPublic3 Estimated Story Points

Description

Background

  • This ticket is for the initial deploy of the ab test on Feb 11

Requirements

Sign off steps

  • Make sure the deployment steps have been documented and the analyst knows which data to ignore as part of the A/B test due to bugs.

Event Timeline

bwang triaged this task as High priority.Jan 17 2025, 3:42 PM
bwang moved this task from Incoming to Sprint Backlog on the Web-Team board.
Jdlrobson changed the task status from Open to In Progress.Jan 17 2025, 4:56 PM
Jdlrobson moved this task from Sprint Backlog to Q3 on the Web-Team board.
Jdlrobson-WMF set the point value for this task to 3.Jan 23 2025, 4:12 PM
Jdlrobson-WMF moved this task from Q3 to Sprint Backlog on the Web-Team board.
Jdlrobson-WMF subscribed.

Per team estimation

Jdrewniak subscribed.
Jdlrobson-WMF updated the task description. (Show Details)

@bwang to address the TBDs.

For tech news: As part of this year's work around improving the ways readers discover content on the wikis, we will be running an experiment with a small number of readers that displays some suggestions for related or interesting articles within the search bar. Please check out our project page for more info.

Deploy dates:

  • Test deploy: Feb 10, or feb 11
  • Deploy: Feb 12, or Feb 17

From refinement meeting:

  • We will begin with deployments to cawiki and euwiki which will help us determine whether any increases or decreases to the sampling rate will be necessary
  • After 2/3 days of data collection, we will continue with a deployment to all wikis

Change #1118566 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[operations/mediawiki-config@master] Deploy web ab test to eu and ca wiki for an initial test

https://gerrit.wikimedia.org/r/1118566

Change #1118566 merged by jenkins-bot:

[operations/mediawiki-config@master] Deploy web ab test to eu and ca wiki for an initial test

https://gerrit.wikimedia.org/r/1118566

Mentioned in SAL (#wikimedia-operations) [2025-02-11T22:33:57Z] <jdrewniak@deploy2002> Started scap sync-world: Backport for [[gerrit:1118566|Deploy web ab test to eu and ca wiki for an initial test (T384019)]]

Mentioned in SAL (#wikimedia-operations) [2025-02-11T22:36:52Z] <jdrewniak@deploy2002> jdrewniak, bwang: Backport for [[gerrit:1118566|Deploy web ab test to eu and ca wiki for an initial test (T384019)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-02-11T22:48:02Z] <jdrewniak@deploy2002> Finished scap sync-world: Backport for [[gerrit:1118566|Deploy web ab test to eu and ca wiki for an initial test (T384019)]] (duration: 14m 05s)

No events being logged to product_metrics_web_base_search_ab_test_clicks superset

There is an issue with the config. I will push up a fix.

Change #1119555 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] Fix name of ABTestEnrollment configuration

https://gerrit.wikimedia.org/r/1119555

Change #1119555 merged by jenkins-bot:

[operations/mediawiki-config@master] Fix name of ABTestEnrollment configuration

https://gerrit.wikimedia.org/r/1119555

Mentioned in SAL (#wikimedia-operations) [2025-02-13T21:28:31Z] <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1119555|Fix name of ABTestEnrollment configuration (T384019)]]

Mentioned in SAL (#wikimedia-operations) [2025-02-13T21:31:15Z] <tgr@deploy2002> jdlrobson, tgr: Backport for [[gerrit:1119555|Fix name of ABTestEnrollment configuration (T384019)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-02-13T21:47:55Z] <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1119555|Fix name of ABTestEnrollment configuration (T384019)]] (duration: 19m 24s)

I managed to test this in browser and could see the events firing. There is a lag so I won't be able to confirm the events got through until approx 6pm PST

I can confirm events are now coming in for click tracking

bwang renamed this task from Deploy Empty Search A/B test to Initial Empty Search A/B test Deployment.Feb 18 2025, 6:08 PM
bwang updated the task description. (Show Details)
bwang updated the task description. (Show Details)

I've updated this task description now that we had our initial deploy last week. I filed T386734 to account for the updates/fixes for the AB test. I think this can be signed off

@bwang can this be moved to sign off? Seems like we finished the initial deploy to cawiki and euwiki.

Suggestion: Perhaps https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/1120641 can be included in this for completeness?

@bwang while doing sign off steps I noticed we have T386734 but not deploy ticket for the actual A/B test. Can you create it (or expand the scope of T386734 to cover the actual deploy)? Or is the plan for deployment to happen in sprint 4?