Page MenuHomePhabricator

Add skin key for mediawiki_web_ab_test_enrollment schema firing events
Closed, ResolvedPublic5 Estimated Story Points

Description

Analyst has reported that when mediawiki_web_ab_test_enrollment schema fires, it does not indicate what skin the user in on.
Even if the test is intended to only run on Vector 2022, analyst has reported that there are exceptions when the AB test schema still fires when user switches skin, which makes it unclear if the AB Test data is firing on the correct environment.

Success Criteria

  • Skin key for mediawiki_web_ab_test_enrollment

Testing

  • Check that the skin name is shown when the mediawiki_web_ab_test_enrollment schema fires

Event Timeline

Copying Slack conversation to here:

Schemas are specific to individual teams' specific data collection needs.
The Metrics Platform schema includes mediawiki.skin and that's something that users will be able to request when they're configuring a stream and subscribing to events emitted by Metrics Platform-based instruments.
The Web team can certainly create a schema fragment that defines some common fields that is then included in Web team's existing schemas such as [[ https://gerrit.wikimedia.org/r/plugins/gitiles/schemas/event/secondary/+/refs/heads/master/jsonschema/analytics/mediawiki/web_ab_test_enrollment/current.yaml | analytics/mediawiki/web_ab_test_enrollment ]]and analytics/mediawiki/web_ui_scroll. For example the Android team has fragment/analytics/mobile_apps/wikipedia_android_app that's used in all their schemas (such as analytics/mobile_apps/android_app_session). In addition to the new/proposed skin I see a few fields that can be factored out of Web's existing schemas into a "mediawiki_web_common" fragment (fields such as wiki, is_anon, is_bot).
But skin should not be a default field for all schemas, as there are many schemas where it doesn't make sense to include that. (edited)

Change 911412 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Creates web schema fragment

https://gerrit.wikimedia.org/r/911412

ovasileva renamed this task from Add skin key for mediawiki_web_ab_test_enrollment schema firing events to [SPIKE] Add skin key for mediawiki_web_ab_test_enrollment schema firing events.Apr 27 2023, 5:10 PM

Change 911412 merged by jenkins-bot:

[schemas/event/secondary@master] Creates web schema fragment

https://gerrit.wikimedia.org/r/911412

Change 916625 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] References new fragment in scroll and editattemptstep

https://gerrit.wikimedia.org/r/916625

KSarabia-WMF renamed this task from [SPIKE] Add skin key for mediawiki_web_ab_test_enrollment schema firing events to Add skin key for mediawiki_web_ab_test_enrollment schema firing events.May 9 2023, 4:47 PM
ovasileva set the point value for this task to 5.May 9 2023, 4:48 PM

Change 919157 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Add new fragment for editattemptstep

https://gerrit.wikimedia.org/r/919157

Change 919214 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] Run A/B test enrollment code for any A/B test

https://gerrit.wikimedia.org/r/919214

Change 919237 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/WikimediaEvents@master] Update webuiscroll.js

https://gerrit.wikimedia.org/r/919237

Change 919214 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Run A/B test enrollment code for any A/B test

https://gerrit.wikimedia.org/r/919214

Change 916625 merged by jenkins-bot:

[schemas/event/secondary@master] References new fragment in scroll

https://gerrit.wikimedia.org/r/916625

Change 919237 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Update webuiscroll.js

https://gerrit.wikimedia.org/r/919237

Change 919879 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Modifies AB Test Enrollment schema

https://gerrit.wikimedia.org/r/919879

Change 919914 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Removes app identifier line

https://gerrit.wikimedia.org/r/919914

Change 919914 merged by jenkins-bot:

[schemas/event/secondary@master] Removes app identifier line

https://gerrit.wikimedia.org/r/919914

Change 919928 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[schemas/event/secondary@master] Web UI Scroll: Use latest web fragment

https://gerrit.wikimedia.org/r/919928

Change 919157 merged by jenkins-bot:

[schemas/event/secondary@master] Add new fragment for editattemptstep

https://gerrit.wikimedia.org/r/919157

Change 919879 merged by jenkins-bot:

[schemas/event/secondary@master] Modifies AB Test Enrollment schema

https://gerrit.wikimedia.org/r/919879

Change 919928 merged by jenkins-bot:

[schemas/event/secondary@master] Web UI Scroll: Use latest web fragment

https://gerrit.wikimedia.org/r/919928

Change 919932 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/WikimediaEvents@master] Update schema fields and corresponding js

https://gerrit.wikimedia.org/r/919932

Jdlrobson subscribed.

Okay. We're gonna want to QA the actual data coming in here on beta cluster and mediawiki.org tomorrow.

Change 919932 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Update schema fields and corresponding js

https://gerrit.wikimedia.org/r/919932

Change 920386 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[operations/mediawiki-config@master] Enable zebra ab test in hewiki

https://gerrit.wikimedia.org/r/920386

Edtadros subscribed.

Test Result - Beta

Status: ✅ PASS
Environment: beta
OS: macOS Ventura
Browser: Chrome
Device: MBP
Emulated Device:NA

Test Artifact(s):

QA Steps

✅ AC1: Check that the skin name is shown when the mediawiki_web_ab_test_enrollment schema fires
Verified that the test/event doesn't appear in Vector Legacy, MinervaNeue, MonoBook, and Timeless.

Screenshot 2023-05-16 at 2.03.40 PM.png (1×1 px, 301 KB)

Kim will verify this in production tomorrow after we deploy to Hebrew Wikipedia.

Change 921059 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[operations/mediawiki-config@master] Reverts hewiki A/B test

https://gerrit.wikimedia.org/r/921059

Change 921059 merged by jenkins-bot:

[operations/mediawiki-config@master] Reverts hewiki A/B test

https://gerrit.wikimedia.org/r/921059

Mentioned in SAL (#wikimedia-operations) [2023-05-18T20:06:11Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:921059|Reverts hewiki A/B test (T335309)]]

Mentioned in SAL (#wikimedia-operations) [2023-05-18T20:07:42Z] <urbanecm@deploy1002> ksarabia and urbanecm: Backport for [[gerrit:921059|Reverts hewiki A/B test (T335309)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-05-18T20:16:36Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:921059|Reverts hewiki A/B test (T335309)]] (duration: 10m 25s)

Have verified that skin field is available in mediawiki_web_ab_test_enrollment schema. The skin name is captured as expected.