Following deployment of the Tone Check AB Test in T387918, we will need to confirm that AB test events are logging as expected and people are being bucketed into the control and test groups in the proportions we expect.
Requirements:
- Confirm that bucket field is populated.
- Confirm anonymous_user_token field is populated for unregistered users included in the test.
- Confirm users are bucketed in the control and test group in expected proportions based on a 50/50 split.
Overview of AB test instrumentation and events:
- AB test data will be logged in EditAttemptStep.
- Test assignments are indicated by the event.bucket field.
[To be further populated with bucketing requirements and instrumentation once defined]
Query to confirm bucket balance [can be run on Superset; adjust to reflect dates of deployment and any other bucketing conditions] :
SELECT wiki, test_group, COUNT(editing_session) AS n_sessions, COUNT(user_id) AS n_user_ids FROM( SELECT event.wiki AS wiki, event.editing_session_id AS editing_session, IF(event.user_id != 0, concat(cast(event.wiki AS VARCHAR), '-', cast(event.user_id AS VARCHAR)), event.anonymous_user_token) as user_id, event.bucket AS test_group, COUNT(1) AS n_events FROM event.editattemptstep WHERE -- update based on date AB test was deployed year = 2025 AND month = 05 AND DAY = 18 -- update based bucket names AND event.bucket IN ('[INSERT TEST NAME]-control', 'INSERT TEST NAME]-test') -- update with partner wikis AND event.wiki IN ('eswiki') GROUP BY event.wiki, event.editing_session_id, IF(event.user_id != 0, concat(cast(event.wiki AS VARCHAR), '-', cast(event.user_id AS VARCHAR)), event.anonymous_user_token), event.bucket ) GROUP BY test_group, wiki