test wiki
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	bwang
	Feb 18 2025, 6:12 PM

Description

Background

This ticket is for the updated deploy of the AB test on Feb 20

Requirements

Feb 20: euwiki, cawiki, testwiki

bucketing rate: 50/50
session length sampling rate : 100%
click tracking sampling rate: 100%

Config updates

Experiment names are updated to be "RelatedArticles test experiment" Done in 0f563d4437
performer_is_temp and mediawiki_database are added in the stream configuration https://gerrit.wikimedia.org/r/c/1120609
We backport fixes for https://phabricator.wikimedia.org/T386229 (wmf/1.44.0-wmf.16) [if deploying https://gerrit.wikimedia.org/r/c/1120609 on Wednesday 19th]

Data QA

Confirm we are seeing an increase in empty_search_suggestion clicks (up from 3).

SELECT count(*) FROM event.product_metrics_web_base_search_ab_test_clicks
WHERE year=2025 and month=2
AND action='click'
AND action_source='empty_search_suggestion'

Sign off steps

Make sure the deployment steps have been documented and the analyst knows which data to ignore as part of the A/B test due to bugs.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Remove init event from Search AB test and also remove ABTestEnrollment.js.	mediawiki/extensions/WikimediaEvents	wmf/1.44.0-wmf.16	+41 -84
	Update Search AB test config, increase bucketing/sampling rates for eu/ca, deploy to testwiki	operations/mediawiki-config	master	+42 -17

Customize query in gerrit

Related Objects

Mentioned In: T384619: Update skins to support different logos at different resolutions
T386735: Show empty search recommendation event is missing funnel data
T384019: Initial Empty Search A/B test Deployment
Mentioned Here: T386495: Fix session tick mixin relating to when events fire
T386735: Show empty search recommendation event is missing funnel data
T386849: Deploy empty search AB test to all wikis
T384619: Update skins to support different logos at different resolutions
T386229: No events being logged to product_metrics.web_base.search_ab_test_session_ticks

Event Timeline

bwang created this task.Feb 18 2025, 6:12 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 18 2025, 6:12 PM

bwang updated the task description. (Show Details)Feb 18 2025, 6:12 PM

bwang mentioned this in T384019: Initial Empty Search A/B test Deployment.Feb 18 2025, 6:15 PM

Change #1120609 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[operations/mediawiki-config@master] Update Search AB test config, increase bucketing/sampling rates for eu/ca, deploy to testwiki

https://gerrit.wikimedia.org/r/1120609

gerritbot added a project: Patch-For-Review.Feb 18 2025, 6:15 PM

bwang triaged this task as High priority.Feb 18 2025, 6:24 PM

Jdrewniak changed the task status from Open to In Progress.Feb 18 2025, 6:58 PM

Jdrewniak assigned this task to bwang.

Jdrewniak moved this task from Incoming to Q3 on the Web-Team board.

It is enough to use configuration here to get these fields to show up, or do we also need to send them in WikimediaEvents ? Have we confirmed the config patch is all we need to do?

@Jdlrobson it should be enough to use the config, just like all the other provide_values fields. i could double check with jennifer that thats enough

bwang updated the task description. (Show Details)Feb 18 2025, 10:29 PM

Change #1120685 had a related patch set uploaded (by Jdlrobson; author: Bernard Wang):

[mediawiki/extensions/WikimediaEvents@wmf/1.44.0-wmf.16] Remove init event from Search AB test and also remove ABTestEnrollment.js.

https://gerrit.wikimedia.org/r/1120685

Jdlrobson-WMF updated the task description. (Show Details)Feb 19 2025, 12:47 AM

Jdlrobson-WMF set the point value for this task to 3.Feb 19 2025, 4:30 PM

Jdlrobson-WMF moved this task from Q3 to Sprint Backlog on the Web-Team board.

Jdlrobson-WMF mentioned this in T386735: Show empty search recommendation event is missing funnel data.Feb 19 2025, 4:58 PM

Jdlrobson-WMF updated the task description. (Show Details)

Jdlrobson-WMF moved this task from Sprint Backlog to Q3 Sprint 3 (Feb 19 - Mar 5, 2025) on the Web-Team board.Feb 19 2025, 6:16 PM

Jdlrobson-WMF edited projects, added Web-Team (Q3 Sprint 3 (Feb 19 - Mar 5, 2025)); removed Web-Team.

Jdlrobson-WMF claimed this task.Feb 19 2025, 6:44 PM

Jdlrobson-WMF moved this task from Ready for development to In progress on the Web-Team (Q3 Sprint 3 (Feb 19 - Mar 5, 2025)) board.

Jdlrobson-WMF updated the task description. (Show Details)

Jdlrobson-WMF added a project: FY2024-25 KR 3.1 Content Discovery.Feb 19 2025, 7:03 PM

Change #1120609 merged by jenkins-bot:

[operations/mediawiki-config@master] Update Search AB test config, increase bucketing/sampling rates for eu/ca, deploy to testwiki

https://gerrit.wikimedia.org/r/1120609

Mentioned in SAL (#wikimedia-operations) [2025-02-19T21:28:07Z] <cjming@deploy2002> Started scap sync-world: Backport for [[gerrit:1119579|Footer: Wikimedia icon should collapse at lower resolutions (T384619)]], [[gerrit:1120609|Update Search AB test config, increase bucketing/sampling rates for eu/ca, deploy to testwiki (T386734)]]

Mentioned in SAL (#wikimedia-operations) [2025-02-19T21:31:10Z] <cjming@deploy2002> jdlrobson, cjming, bwang: Backport for [[gerrit:1119579|Footer: Wikimedia icon should collapse at lower resolutions (T384619)]], [[gerrit:1120609|Update Search AB test config, increase bucketing/sampling rates for eu/ca, deploy to testwiki (T386734)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-02-19T21:39:55Z] <cjming@deploy2002> Finished scap sync-world: Backport for [[gerrit:1119579|Footer: Wikimedia icon should collapse at lower resolutions (T384619)]], [[gerrit:1120609|Update Search AB test config, increase bucketing/sampling rates for eu/ca, deploy to testwiki (T386734)]] (duration: 11m 47s)

Jdrewniak claimed this task.Feb 19 2025, 11:39 PM

I've taken a look at the data and I can verify the following:

Sampling rate has increased to 100%
- there are many more events coming in, e.g.

Autocomplete_search_suggestion 211
Empty_search_suggestion 22
Search_box 565

Bucketing is 50/50 and the experiment is renamed to RelatedArticles test experiment

{ "RelatedArticles test experiment" : "experimentEnabled" } 3564
{ "RelatedArticles test experiment" : "experimentDisabled" } 3465

performer_is_temp and mediawiki_database have been added to the data. It looks like performer.is_temp is null for anons, and either true or false when performer.is_logged_in is true.
Mediawiki_database is present - values are either euwiki or cawiki.

Make sure the deployment steps have been documented and the analyst knows which data to ignore as part of the A/B test due to bugs.

The general deployment task is T386849. There are still some outstanding bugs, T386735, T386495, so @jwang I think all this data should still be ignored until the test is widely deployed next week.

Jdrewniak updated the task description. (Show Details)Feb 20 2025, 5:43 PM

The general deployment task is T386849. There are still some outstanding bugs, T386735, T386495, so @jwang I think all this data should still be ignored until the test is widely deployed next week.

As I understand T386495 doesnt greatly impact the data analysis, its just a fix we want before the final deploy to make things easier. T386495 was backported too, and i can verify the fix on test.wiki. So I believe the data starting yesterday is good for @jwang to QA and analyze.

I'm going to sign off this task too given the backports all are confirmed and verified in production!

bwang closed this task as Resolved.Feb 21 2025, 9:38 PM

Change #1120685 abandoned by Jdlrobson: