mediawiki_web_ab_test_enrollment schema is logging users in the unsampled bucket
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	Jdlrobson
	Dec 13 2021, 11:40 PM

Description

mediawiki_web_ab_test_enrollment logs users in the A/B test.

30% of people should be in the control bucket
30% of people should be in the stickyHeaderDisabled bucket
30% of people should be in the stickyHeaderEnabled bucket
10% of people should be in the unsampled bucket and not appear in the database.

However, in current data on test wikipedia we are seeing users in the unsampled bucket appear in the database (Hue query select * from mediawiki_web_ab_test_enrollment WHERE year = 2021 AND wiki = 'testwiki')

TODO

Don't log user in the unsampled bucket
Add validation to Vector to deal with bad config

Details

Other Assignee: Jdlrobson

Subject	Repo	Branch	Lines +/-
Validate the value of VectorWebABTestEnrollment	mediawiki/skins/Vector	master	+184 -3
Prevent A/B test enrollment hook from firing for unsampled	mediawiki/skins/Vector	wmf/1.38.0-wmf.12	+1 -1
Prevent A/B test enrollment hook from firing for unsampled	mediawiki/skins/Vector	master	+1 -1

Customize query in gerrit

Related Objects

Mentioned Here: T292587: Sticky header: Create A/B test schema and tie to sticky header feature
rSVEC62e84e7467c1: Prevent A/B test enrollment hook from firing for unsampled

Event Timeline

Jdlrobson created this task.Dec 13 2021, 11:40 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 13 2021, 11:40 PM

Jdlrobson triaged this task as High priority.Dec 13 2021, 11:40 PM

Debugged this in Slack with Clare. Looks like we need to add a conditional.

Change 746973 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/skins/Vector@master] Prevent A/B test enrollment hook from firing for unsampled

https://gerrit.wikimedia.org/r/746973

gerritbot added a project: Patch-For-Review.Dec 14 2021, 12:32 AM

Change 746973 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Prevent A/B test enrollment hook from firing for unsampled

https://gerrit.wikimedia.org/r/746973

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.16; 2022-01-03).Dec 14 2021, 2:00 AM

Maintenance_bot removed a project: Patch-For-Review.Dec 14 2021, 2:10 AM

Jdlrobson updated the task description. (Show Details)Dec 14 2021, 6:13 PM

LGoto moved this task from Upcoming to Ready for Development on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.Dec 14 2021, 6:15 PM

LGoto set the point value for this task to 3.

Change 747075 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[mediawiki/skins/Vector@wmf/1.38.0-wmf.12] Prevent A/B test enrollment hook from firing for unsampled

https://gerrit.wikimedia.org/r/747075

gerritbot added a project: Patch-For-Review.Dec 14 2021, 6:43 PM

Jdlrobson moved this task from Ready for Development to Doing on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.Dec 14 2021, 7:12 PM

Jdlrobson updated Other Assignee, added: Jdlrobson.

Change 747075 merged by jenkins-bot:

[mediawiki/skins/Vector@wmf/1.38.0-wmf.12] Prevent A/B test enrollment hook from firing for unsampled

https://gerrit.wikimedia.org/r/747075

Change 747197 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] Validate the value of VectorWebABTestEnrollment

https://gerrit.wikimedia.org/r/747197

Jdlrobson moved this task from Doing to Code Review on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.Dec 14 2021, 7:44 PM

Mentioned in SAL (#wikimedia-operations) [2021-12-14T19:53:00Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.12/skins/Vector/resources/skins.vector.es6/AB.js: 62e84e7467c1765986cd1f80b466b8cacc6d91f6: Prevent A/B test enrollment hook from firing for unsampled (T297662) (duration: 00m 56s)