Page MenuHomePhabricator
Create Task
Maniphest T335309

Add skin key for mediawiki_web_ab_test_enrollment schema firing events
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Analyst has reported that when mediawiki_web_ab_test_enrollment schema fires, it does not indicate what skin the user in on.
Even if the test is intended to only run on Vector 2022, analyst has reported that there are exceptions when the AB test schema still fires when user switches skin, which makes it unclear if the AB Test data is firing on the correct environment.

Success Criteria

  • Skin key for mediawiki_web_ab_test_enrollment

Testing

  • Check that the skin name is shown when the mediawiki_web_ab_test_enrollment schema fires

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Reverts hewiki A/B testoperations/mediawiki-configmaster+1 -12
Update schema fields and corresponding jsmediawiki/extensions/WikimediaEventsmaster+7 -8
Web UI Scroll: Use latest web fragmentschemas/event/secondarymaster+187 -4
Modifies AB Test Enrollment schemaschemas/event/secondarymaster+159 -3
Add new fragment for editattemptstepschemas/event/secondarymaster+603 -3
Creates web schema fragmentschemas/event/secondarymaster+195 -0
Removes app identifier lineschemas/event/secondarymaster+135 -4
Update webuiscroll.jsmediawiki/extensions/WikimediaEventsmaster+59 -17
References new fragment in scrollschemas/event/secondarymaster+215 -3
Run A/B test enrollment code for any A/B testmediawiki/skins/Vectormaster+48 -33
Customize query in gerrit

Related Objects
Search...

Event Timeline

KSarabia-WMF created this task.Apr 24 2023, 4:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 24 2023, 4:49 PM
ovasileva triaged this task as High priority.Apr 24 2023, 6:05 PM
ovasileva added a parent task: T333493: Setup A/B test requirement for Zebra.
ovasileva moved this task from Incoming to Not ready to estimate on the Web-Team-Backlog-Archived board.
KSarabia-WMF added a comment.Apr 24 2023, 7:33 PM

Copying Slack conversation to here:

Schemas are specific to individual teams' specific data collection needs.
The Metrics Platform schema includes mediawiki.skin and that's something that users will be able to request when they're configuring a stream and subscribing to events emitted by Metrics Platform-based instruments.
The Web team can certainly create a schema fragment that defines some common fields that is then included in Web team's existing schemas such as [[ https://gerrit.wikimedia.org/r/plugins/gitiles/schemas/event/secondary/+/refs/heads/master/jsonschema/analytics/mediawiki/web_ab_test_enrollment/current.yaml | analytics/mediawiki/web_ab_test_enrollment ]]and analytics/mediawiki/web_ui_scroll. For example the Android team has fragment/analytics/mobile_apps/wikipedia_android_app that's used in all their schemas (such as analytics/mobile_apps/android_app_session). In addition to the new/proposed skin I see a few fields that can be factored out of Web's existing schemas into a "mediawiki_web_common" fragment (fields such as wiki, is_anon, is_bot).
But skin should not be a default field for all schemas, as there are many schemas where it doesn't make sense to include that. (edited)

gerritbot added a comment.Apr 24 2023, 9:58 PM

Change 911412 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Creates web schema fragment

https://gerrit.wikimedia.org/r/911412

gerritbot added a project: Patch-For-Review.Apr 24 2023, 9:58 PM
ovasileva renamed this task from Add skin key for mediawiki_web_ab_test_enrollment schema firing events to [SPIKE] Add skin key for mediawiki_web_ab_test_enrollment schema firing events.Apr 27 2023, 5:10 PM
gerritbot added a comment.May 1 2023, 11:08 PM

Change 911412 merged by jenkins-bot:

[schemas/event/secondary@master] Creates web schema fragment

https://gerrit.wikimedia.org/r/911412

Maintenance_bot removed a project: Patch-For-Review.May 1 2023, 11:10 PM
gerritbot added a comment.May 5 2023, 10:16 PM

Change 916625 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] References new fragment in scroll and editattemptstep

https://gerrit.wikimedia.org/r/916625

gerritbot added a project: Patch-For-Review.May 5 2023, 10:16 PM
ovasileva edited projects, added Web-Team-Backlog-Archived (Web Team FY2022-23 Q4 Sprint 3); removed Web-Team-Backlog-Archived.May 9 2023, 4:44 PM
KSarabia-WMF renamed this task from [SPIKE] Add skin key for mediawiki_web_ab_test_enrollment schema firing events to Add skin key for mediawiki_web_ab_test_enrollment schema firing events.May 9 2023, 4:47 PM
ovasileva set the point value for this task to 5.May 9 2023, 4:48 PM
Jdlrobson moved this task from Incoming to Doing on the Web-Team-Backlog-Archived (Web Team FY2022-23 Q4 Sprint 3) board.May 9 2023, 8:38 PM
gerritbot added a comment.May 11 2023, 4:03 PM

Change 919157 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Add new fragment for editattemptstep

https://gerrit.wikimedia.org/r/919157

Jdlrobson moved this task from Doing to Code Review on the Web-Team-Backlog-Archived (Web Team FY2022-23 Q4 Sprint 3) board.May 11 2023, 4:54 PM
gerritbot added a comment.May 11 2023, 7:04 PM

Change 919214 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] Run A/B test enrollment code for any A/B test

https://gerrit.wikimedia.org/r/919214

gerritbot added a comment.May 11 2023, 9:36 PM

Change 919237 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/WikimediaEvents@master] Update webuiscroll.js

https://gerrit.wikimedia.org/r/919237

gerritbot added a comment.May 11 2023, 10:15 PM

Change 919214 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Run A/B test enrollment code for any A/B test

https://gerrit.wikimedia.org/r/919214

gerritbot added a comment.May 11 2023, 11:03 PM

Change 916625 merged by jenkins-bot:

[schemas/event/secondary@master] References new fragment in scroll

https://gerrit.wikimedia.org/r/916625

Jdlrobson moved this task from Code Review to Needs More Work on the Web-Team-Backlog-Archived (Web Team FY2022-23 Q4 Sprint 3) board.May 12 2023, 7:46 PM
gerritbot added a comment.May 15 2023, 6:56 PM

Change 919237 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Update webuiscroll.js

https://gerrit.wikimedia.org/r/919237

gerritbot added a comment.May 15 2023, 6:56 PM

Change 919879 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Modifies AB Test Enrollment schema

https://gerrit.wikimedia.org/r/919879

Jdlrobson moved this task from Needs More Work to Doing on the Web-Team-Backlog-Archived (Web Team FY2022-23 Q4 Sprint 3) board.May 15 2023, 7:22 PM
gerritbot added a comment.May 15 2023, 9:36 PM

Change 919914 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[schemas/event/secondary@master] Removes app identifier line

https://gerrit.wikimedia.org/r/919914

gerritbot added a comment.May 15 2023, 9:50 PM

Change 919914 merged by jenkins-bot:

[schemas/event/secondary@master] Removes app identifier line

https://gerrit.wikimedia.org/r/919914

gerritbot added a comment.May 15 2023, 10:46 PM

Change 919928 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[schemas/event/secondary@master] Web UI Scroll: Use latest web fragment

https://gerrit.wikimedia.org/r/919928

gerritbot added a comment.May 16 2023, 12:10 AM

Change 919157 merged by jenkins-bot:

[schemas/event/secondary@master] Add new fragment for editattemptstep

https://gerrit.wikimedia.org/r/919157

gerritbot added a comment.May 16 2023, 12:10 AM

Change 919879 merged by jenkins-bot:

[schemas/event/secondary@master] Modifies AB Test Enrollment schema

https://gerrit.wikimedia.org/r/919879

gerritbot added a comment.May 16 2023, 12:19 AM

Change 919928 merged by jenkins-bot:

[schemas/event/secondary@master] Web UI Scroll: Use latest web fragment

https://gerrit.wikimedia.org/r/919928

Jdlrobson moved this task from Doing to Code Review on the Web-Team-Backlog-Archived (Web Team FY2022-23 Q4 Sprint 3) board.May 16 2023, 12:27 AM
Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 12:30 AM
gerritbot added a comment.May 16 2023, 12:52 AM

Change 919932 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[mediawiki/extensions/WikimediaEvents@master] Update schema fields and corresponding js

https://gerrit.wikimedia.org/r/919932

gerritbot added a project: Patch-For-Review.May 16 2023, 12:52 AM
Jdlrobson reassigned this task from KSarabia-WMF to Edtadros.May 16 2023, 1:11 AM
Jdlrobson moved this task from Code Review to QA on the Web-Team-Backlog-Archived (Web Team FY2022-23 Q4 Sprint 3) board.
Jdlrobson subscribed.

Okay. We're gonna want to QA the actual data coming in here on beta cluster and mediawiki.org tomorrow.

gerritbot added a comment.May 16 2023, 1:19 AM

Change 919932 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Update schema fields and corresponding js

https://gerrit.wikimedia.org/r/919932

Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 1:29 AM
ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.10; 2023-05-23).May 16 2023, 8:01 PM
gerritbot added a comment.May 16 2023, 10:31 PM

Change 920386 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[operations/mediawiki-config@master] Enable zebra ab test in hewiki

https://gerrit.wikimedia.org/r/920386

gerritbot added a project: Patch-For-Review.May 16 2023, 10:31 PM
Edtadros removed Edtadros as the assignee of this task.May 16 2023, 10:35 PM
Edtadros moved this task from QA to Ready for Signoff on the Web-Team-Backlog-Archived (Web Team FY2022-23 Q4 Sprint 3) board.
Edtadros subscribed.

Test Result - Beta

Status: ✅ PASS
Environment: beta
OS: macOS Ventura
Browser: Chrome
Device: MBP
Emulated Device:NA

Test Artifact(s):

QA Steps

✅ AC1: Check that the skin name is shown when the mediawiki_web_ab_test_enrollment schema fires
Verified that the test/event doesn't appear in Vector Legacy, MinervaNeue, MonoBook, and Timeless.

Screenshot 2023-05-16 at 2.03.40 PM.png (1×1 px, 301 KB)

Jdlrobson assigned this task to KSarabia-WMF.May 17 2023, 1:56 AM
Jdlrobson moved this task from Ready for Signoff to QA in Production on the Web-Team-Backlog-Archived (Web Team FY2022-23 Q4 Sprint 3) board.

Kim will verify this in production tomorrow after we deploy to Hebrew Wikipedia.

gerritbot added a comment.May 18 2023, 3:58 PM

Change 921059 had a related patch set uploaded (by Kimberly Sarabia; author: Kimberly Sarabia):

[operations/mediawiki-config@master] Reverts hewiki A/B test

https://gerrit.wikimedia.org/r/921059

KSarabia-WMF updated the task description. (Show Details)May 18 2023, 5:32 PM
gerritbot added a comment.May 18 2023, 8:04 PM

Change 921059 merged by jenkins-bot:

[operations/mediawiki-config@master] Reverts hewiki A/B test

https://gerrit.wikimedia.org/r/921059

Stashbot added a comment.May 18 2023, 8:06 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-18T20:06:11Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:921059|Reverts hewiki A/B test (T335309)]]

Stashbot added a comment.May 18 2023, 8:07 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-18T20:07:42Z] <urbanecm@deploy1002> ksarabia and urbanecm: Backport for [[gerrit:921059|Reverts hewiki A/B test (T335309)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet

Stashbot added a comment.May 18 2023, 8:16 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-18T20:16:36Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:921059|Reverts hewiki A/B test (T335309)]] (duration: 10m 25s)

KSarabia-WMF updated the task description. (Show Details)May 22 2023, 5:24 PM
jwang added a comment.May 22 2023, 5:27 PM

Have verified that skin field is available in mediawiki_web_ab_test_enrollment schema. The skin name is captured as expected.

KSarabia-WMF closed this task as Resolved.May 22 2023, 8:47 PM
jwang mentioned this in T333222: Create instrumentation spec for Zebra #9 prototype.May 30 2023, 9:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits