Schema QA : mediawiki_web_ab_test_enrollment
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jwang
	Dec 16 2021, 1:02 AM

Description

As one of the instrumentations for sticky header, Web team has deployed the instrumentation to record AB test group enrollment. T292587

The related events will be stored in a new schema mediawiki_web_ab_test_enrollment. Sample rate is 100%

What has been checked	Status	Note
Check experiment group assignment	✅pass	Schema captured three groups: control , stickyHeaderDisabled, stickyHeaderEnabled
Check daily pageviews and sessions by groups	✅pass	The schema runs on every page view for users that are inside the A/B test while the A/B test is running
Check bucketing	✅pass	The distribution ratio of control, stickyHeaderDisabled and stickyHeaderEnabled groups is close to 1:1:1
Check whether `mediawiki_web_ab_test_enrollment` can join with click events on stiky header	✅pass	Two schemas can be joined by condition `DesktopWebUIActionsTracking.event.token` =`mediawiki_web_ab_test_enrollment.web_session_id`
Check whether `mediawiki_web_ab_test_enrollment` can join with scroll events	✅pass	Two scehemas can be joined by `web_session_id` . We only see scroll-back event in stickyHeaderDisabled group, not in stickyHeaderEnabled group. Maybe it indicates that sticky header reduced scroll-back on 12-28-2021. We needs more data to confirm after we collect data from AB test

Please see QA notebook for details.

Issues	Status	Note
unsampled group is recorded in the data	✅Engineer has fixed it	Data since 12-15-2021 is corrected
duplicated records on the same web_session_id	✅ Confirmed with engineer that it's expected	T292587#7576512

Status	Assigned	Task
Resolved	ovasileva	T283505 [EPIC] Fixed "sticky" site header
Resolved	• alexhollender_WMF	T292304 [Goal] Deploy sticky header to pilot wikis
Resolved	jwang	T294639 Schema QA sticky header instrumentation
Resolved	jwang	T297852 Schema QA : mediawiki_web_ab_test_enrollment

Explored data and found 2 possible issues.

There are duplicated records on the same web_session_id.
- Status: confirming with engineer whether it's expected. T292587#7574129

jwang updated the task description. (Show Details)

ovasileva triaged this task as High priority.Dec 17 2021, 3:08 PM

@ovasileva I completed post-deployment QA of the instrumentation added to track AB test enrollment. All events appear to be recording as expected.

What has been checked	Status	Note
check experiment group assignment	✅pass	Schema captured three groups: control , stickyHeaderDisabled, stickyHeaderEnabled
Check daily pageviews and sessions by groups	✅pass	The schema runs on every page view for users that are inside the A/B test while the A/B test is running
Check bucketing	✅pass	The distribution ratio of control, stickyHeaderDisabled and stickyHeaderEnabled groups is close to 1:1:1
Check whether `mediawiki_web_ab_test_enrollment` can join with click events on stiky header	✅pass	Two schemas can be joined by condition `DesktopWebUIActionsTracking.event.token` =`mediawiki_web_ab_test_enrollment.web_session_id`
Check whether `mediawiki_web_ab_test_enrollment` can join with scroll events	✅pass	Two scehemas can be joined by `web_session_id` . We only see scroll-back event in stickyHeaderDisabled group, not in stickyHeaderEnabled group. Maybe it indicates that sticky header reduced scroll-back on 12-28-2021. We needs more data to confirm after we collect data from AB test

Please see QA notebook for details.

Thanks @jwang, resolving!