Page MenuHomePhabricator
Create Task
Maniphest T380926

Configure new streams for the mobile search A/B test
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Description:
We need to create 2 new streams to validate events generated by the session tick mixin and click tracking. We can use the existing product web base schema, which has the right data shape for what we need.

This is instructions on how to configure a new stream.
https://wikitech.wikimedia.org/wiki/Metrics_Platform/Stream_configuration

The new streams will be called:
'product_metrics.web_base.search_ab_test_session_ticks',
'product_metrics.web_base.search_ab_test_clicks',

User Story
As an engineer working on event instrumentation, I need a well-defined schema and stream to validate session tick mixin events, so that event data is reliable and adheres to expected standards.

Requirements
This stream has the right shape for all the ab test click and session tick events

  • after the stream is configured it should show up in Hive

Acceptance Criteria
Stream is configured and validated .

Communication Criteria

Rollback Plan
If issues arise, revert instrumentation changes to the previous state, and disable schema validation temporarily until resolved.

Details

Other Assignee
KSarabia-WMF
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Configure streams for web empty search AB testBernard Wangoperations/mediawiki-configmaster+40 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

KSarabia-WMF created this task.Nov 26 2024, 9:38 PM
Jdrewniak renamed this task from Create/Update Schema for Validating Events from Session Tick Mixin to Create/Update schema for mobile search A/B test .Nov 26 2024, 9:46 PM
Jdrewniak triaged this task as High priority.
Jdrewniak removed a project: Web-Team-Backlog-Archived (FY2024-25 Q2 Sprint 4).
Jdrewniak moved this task from Incoming to Sprint Backlog on the Web-Team board.
Jdrewniak mentioned this in T378109: Mobile search A/B test instrumentation - technical enablers.Nov 26 2024, 9:52 PM
bwang moved this task from Sprint Backlog to Incoming on the Web-Team board.Nov 26 2024, 11:00 PM
bwang changed the task status from Open to In Progress.Nov 27 2024, 5:17 PM
Jdrewniak subscribed.Dec 2 2024, 3:56 PM

After consulting with Metrics Platform regarding the schema, based on the following specification, it looks like we should be able to use the analytics/product_metrics/web/base schema for this experiment. The ui actions can be sent to the action property and session id has an equivalent session_id field.

The test buckets (test & control) can be sent to the experiments fragment which has a assigned property, and the name of the experiment can be logged to the enrolled property.

The search_session_id can be logged as the action_context of each UI action.

Jdrewniak mentioned this in T378115: Implement A/B test bucketing for mobile search recommendation.Dec 2 2024, 4:02 PM
Jdlrobson renamed this task from Create/Update schema for mobile search A/B test to Create/Update SessionTick schema for mobile search A/B test .Dec 4 2024, 5:53 PM
KSarabia-WMF added a subscriber: jwang.Dec 4 2024, 8:26 PM
bwang mentioned this in T381604: Wire up new sessionLength instrumentation in search recommendations.Dec 5 2024, 6:04 PM
bwang subscribed.

My recommendation is to close this ticket, given we can just use the existing analytics/product_metrics/web/base schema. I think we should have this work encompassed in https://phabricator.wikimedia.org/T381604

bwang closed this task as Declined.Dec 5 2024, 6:05 PM
bwang reopened this task as Open.Jan 9 2025, 7:28 PM
Jdlrobson changed the task status from Open to In Progress.Jan 10 2025, 1:06 AM
Jdlrobson assigned this task to bwang.
bwang moved this task from Incoming to Sprint Backlog on the Web-Team board.Jan 10 2025, 5:08 PM
ovasileva added a parent task: T378094: [Epic] A/B test search suggestions on mobile.Jan 13 2025, 11:24 AM
Jdlrobson moved this task from Sprint Backlog to Incoming on the Web-Team board.Jan 13 2025, 5:32 PM
bwang renamed this task from Create/Update SessionTick schema for mobile search A/B test to Create a new stream for the mobile search A/B test .Jan 13 2025, 5:37 PM
bwang updated the task description. (Show Details)
Jdlrobson moved this task from Incoming to Q3 on the Web-Team board.Jan 13 2025, 6:02 PM
bwang renamed this task from Create a new stream for the mobile search A/B test to Configure new stream for the mobile search A/B test .Jan 13 2025, 7:17 PM
bwang mentioned this in T383611: Create new click tracking instrumentation for search recomm A/B test.Jan 13 2025, 7:23 PM
Jdlrobson set the point value for this task to 3.Jan 14 2025, 5:23 PM
Jdlrobson subscribed.

Will require a backport and might require coordination with data engineering.

Jdlrobson moved this task from Q3 to Sprint Backlog on the Web-Team board.Jan 14 2025, 5:25 PM
Jdlrobson moved this task from Sprint Backlog to FY2024-25 Q2 Sprint 6 (Dec 16 - Jan 13, 2025) on the Web-Team board.Jan 14 2025, 5:45 PM
Jdlrobson edited projects, added: Web-Team (FY2024-25 Q2 Sprint 6 (Dec 16 - Jan 13, 2025)); removed: Web-Team.
Jdlrobson moved this task from FY2024-25 Q2 Sprint 6 (Dec 16 - Jan 13, 2025) to Q3 Sprint 1 (Jan 14 - Jan 24, 2025) on the Web-Team board.Jan 14 2025, 5:50 PM
Jdlrobson edited projects, added: Web-Team (Q3 Sprint 1 (Jan 14 - Jan 24, 2025)); removed: Web-Team (FY2024-25 Q2 Sprint 6 (Dec 16 - Jan 13, 2025)).
bwang moved this task from Ready for Development to In Progress on the Web-Team (Q3 Sprint 1 (Jan 14 - Jan 24, 2025)) board.Jan 14 2025, 6:12 PM
Jdlrobson updated Other Assignee, added: KSarabia-WMF.Jan 16 2025, 6:58 PM

A patch will be up shortly and we will review and deploy it in the web team window.

gerritbot added a comment.Jan 16 2025, 7:04 PM

Change #1112074 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[operations/mediawiki-config@master] Configure streams for web empty search AB test

https://gerrit.wikimedia.org/r/1112074

gerritbot added a project: Patch-For-Review.Jan 16 2025, 7:04 PM
bwang renamed this task from Configure new stream for the mobile search A/B test to Configure new streams for the mobile search A/B test .Jan 16 2025, 7:46 PM
bwang updated the task description. (Show Details)
gerritbot added a comment.Jan 16 2025, 10:28 PM

Change #1112074 merged by jenkins-bot:

[operations/mediawiki-config@master] Configure streams for web empty search AB test

https://gerrit.wikimedia.org/r/1112074

Stashbot added a comment.Jan 16 2025, 10:28 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-16T22:28:43Z] <toyofuku@deploy2002> Started scap sync-world: Backport for [[gerrit:1112074|Configure streams for web empty search AB test (T380926)]]

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2025, 10:30 PM
Stashbot added a comment.Jan 16 2025, 10:33 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-16T22:33:24Z] <toyofuku@deploy2002> bwang, toyofuku: Backport for [[gerrit:1112074|Configure streams for web empty search AB test (T380926)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Jan 16 2025, 10:45 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-16T22:45:31Z] <toyofuku@deploy2002> Finished scap sync-world: Backport for [[gerrit:1112074|Configure streams for web empty search AB test (T380926)]] (duration: 16m 47s)

Jdlrobson added a comment.Jan 17 2025, 12:43 AM

The work here should be done, but can't be signed off until T383358 is complete.

Jdlrobson moved this task from In Progress to Sign-Off on the Web-Team (Q3 Sprint 1 (Jan 14 - Jan 24, 2025)) board.Jan 17 2025, 12:43 AM
bwang reassigned this task from bwang to jwang.Jan 17 2025, 3:38 PM
bwang removed jwang as the assignee of this task.
Jdlrobson-WMF claimed this task.Jan 21 2025, 6:50 PM
Jdlrobson-WMF closed this task as Resolved.Jan 24 2025, 10:19 PM

QA to be done as part of T383358

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits