As one of the instrumentations for ToC ( table of content), Web team will deploy the instrumentation to enable AB test on the table of content. (T302046)
== Instrumentation note
The test group enrollment events are stored in `mediawiki_web_ab_test_enrollment` schema.
As of 2022-04-27, user bucketing is based on what page user lands on first, and then persistent for that session. The page_id of the landing page is the salt to decide which test group the session will be in. Session_id is used to identify user session, stored in field `web_session_id`.
**Test population**: logged in users and anonymous users
**Group split**: Sample entire population, 50% to control group, 50% to test group.
== QA summary
| What has been checked | Status | Note |
|---------------------------------------------------------------------------------------------|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| wiki | ✅pass | Enabled on euwiki and hewiki since 2022-04-26|
| Check experiment group assignment | ✅pass | Schema captured two groups: control , treatment , with experiment_name='skin-vector-toc-experiment' |
| Check daily pageviews and sessions by groups | ✅pass | The schema runs on every page view for users that are inside the A/B test while the A/B test is running |
| Check bucketing | ✅pass | The number of unique sessions in control and treatment is close to 1:1 |
| Check whether `mediawiki_web_ab_test_enrollment` can join with click events on stiky header | ✅pass | Two schemas can be joined by condition `DesktopWebUIActionsTracking.event.token` =`mediawiki_web_ab_test_enrollment.web_session_id` and event.name = 'ui.toc' |
| Check whether `mediawiki_web_ab_test_enrollment` can join with scroll events | ✅pass | Two schemas can be joined by `web_session_id `. |
| Check whether `mediawiki_web_ab_test_enrollment` can join with reading depth events | ✅pass | Two schemas can be joined by ` mediawiki_reading_depth.session_token` = `mediawiki_web_ab_test_enrollment.web_session_id` |
| Check the sessions in both treatment and control groups| ❓| 26% of sessions are in two groups on euwiki, 35% of sessions are in tow groups on hewiki |
| Events by spiders| ✅pass | Identified 31 sessions are from Spider on hewiki, 0 from euwiki.|
=== Bugs/Potential Issues
|Issue | Status | Note |
|----|-----|-----|
|web_pageview_id is NULL |❓ | In current AB test instrument, we randomize sessions by the page_id of the landing page. Should we also record the page_id for the purpose of data sanity check and analysis? |
| Check the sessions in both treatment and control groups| ❓| 26% sessions on euwiki are in two groups , 35% sessions on hewiki are in two groups |