Page MenuHomePhabricator
Create Task
Maniphest T378094

[Epic] A/B test search suggestions on mobile
Closed, ResolvedPublic
Actions

Description

Background

We'd like to A/B test the search suggestions on mobile

User story

  • As a product team, we want to a/b test search suggestions on mobile so that we can determine whether they're a useful feature for readers that can affect reader retention

Requirements

  • Add task requirements. Requirements should be user-centric, well-defined, unambiguous, implementable, testable, consistent, and comprehensive

BDD

  • For QA engineer to fill out

Test Steps

  • For QA engineer to fill out

Design

  • Add mockups and design requirements

Acceptance criteria

  • Add acceptance criteria

Communication criteria - does this need an announcement or discussion?

  • Add communication criteria

Rollback plan

  • What is the rollback plan in production for this task if something goes wrong?

This task was created by Version 1.2.0 of the Web team task template using phabulous

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJdrewniakT379707 [Hypothesis] [3.1.8] Content Discovery: Build Feature Additional Article Recommendations (Search A/B test and Article-recommendations A/B test)
 ResolvedJdrewniakT378094 [Epic] A/B test search suggestions on mobile
 ResolvedovasilevaT378095 [Spike] Do people use the search within page button
 ResolvedovasilevaT378109 Mobile search A/B test instrumentation - technical enablers
 ResolvedJdrewniakT378103 Search recommendations A/B test: Write recommendations instrumentation spec
 ResolvedSpikeovasilevaT378101 [Spike] Decide where recommendations code should live
 ResolvedovasilevaT378106 Decide on communication strategy for A/B test
 ResolvedovasilevaT378100 [Spike] Functional + Design comparison of Typeahead search vs Mobile Search
 ResolvedovasilevaT378115 Implement A/B test bucketing for mobile search recommendation
 ResolvedSpikeNBaca-WMFT380630 [Spike] Investigate Stable User Identification for Logged-Out A/B Testing
 ResolvedJdlrobson-WMFT380926 Configure new streams for the mobile search A/B test
ResolvedjwangT387150 Data QA for instrumentation of empty search recommendation AB test
 ResolvedJScherer-WMFT378117 Search A/B test: Add MoreLike-based article suggestions when activating search bar in mobile
 ResolvedSpikeovasilevaT378118 [Spike] Check with Search Platform about traffic impact of recommendations A/B test
 ResolvedovasilevaT380596 Help with Legal and Security Review for New Mixin in Session Metrics Framework
 ResolvedJdlrobson-WMFT381604 Wire up new sessionLength instrumentation in search recommendations
 DeclinedbwangT381616 Update Wikimedia Events to send experiment field to product_metrics schema
 ResolvedJdlrobson-WMFT383266 Refine and document instrumentation spec and process with Jennifer
 ResolvedJdlrobson-WMFT382083 Modify how we bucket users to be based on localStorage rather than sessionStorage
 ResolvedJdlrobson-WMFT383422 Implement experimental sessionLength instrument for use in Search Recommendations A/B test
 ResolvedJdlrobson-WMFT383358 Finish wiring up search recommendations A/B test click tracking
 ResolvedJdlrobson-WMFT383611 Create new click tracking instrumentation for search recomm A/B test
 ResolvedJdlrobson-WMFT382080 Search recommendation clicks should trigger events
 ResolvedNoneT385309 Verify search recommendations A/B test click tracking is working as expected
 ResolvedJdlrobson-WMFT383936 Implement search activity id for Empty Search A/B test click tracking
 OpenDesignJScherer-WMFT381345 [DESIGN] Auto-highlight the first recommendation in the empty state of search
 ResolvedJdlrobson-WMFT384019 Initial Empty Search A/B test Deployment
 ResolvedbwangT386142 Analysis of the deployment plan for the Empty Search A/B test
 ResolvedSToyofuku-WMFT386735 Show empty search recommendation event is missing funnel data
 ResolvedbwangT386495 Fix session tick mixin relating to when events fire
 ResolvedBUG REPORTovasilevaT386849 Deploy empty search AB test to all wikis
 ResolvedEdtadrosT387400 SessionTick instrument should use sessionStorage instead of localStorage
 ResolvedjwangT389326 Analyze results of search suggestions A/B test

Event Timeline

ovasileva created this task.Oct 24 2024, 3:10 PM
Restricted Application added a project: Epic. · View Herald TranscriptOct 24 2024, 3:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ovasileva triaged this task as High priority.Oct 24 2024, 3:10 PM
ovasileva edited projects, added Web-Team-Backlog-Archived; removed Web-Team-Backlog-Archived (FY2024-25 Q2 Sprint 4).
ovasileva moved this task from Incoming to Epics/Goals on the Web-Team-Backlog-Archived board.
ovasileva added a subtask: T378095: [Spike] Do people use the search within page button.Oct 24 2024, 3:14 PM
NBaca-WMF added a subtask: T378109: Mobile search A/B test instrumentation - technical enablers.Oct 24 2024, 4:30 PM
NBaca-WMF added a subtask: T378101: [Spike] Decide where recommendations code should live.
NBaca-WMF added a subtask: T378103: Search recommendations A/B test: Write recommendations instrumentation spec.
NBaca-WMF added a subtask: T378106: Decide on communication strategy for A/B test.
NBaca-WMF added a subtask: T378100: [Spike] Functional + Design comparison of Typeahead search vs Mobile Search.
NBaca-WMF added a subtask: T378115: Implement A/B test bucketing for mobile search recommendation.Oct 24 2024, 4:58 PM
NBaca-WMF added a subtask: T378117: Search A/B test: Add MoreLike-based article suggestions when activating search bar in mobile.Oct 24 2024, 5:17 PM
NBaca-WMF added a subtask: T378118: [Spike] Check with Search Platform about traffic impact of recommendations A/B test.Oct 24 2024, 5:23 PM
NBaca-WMF added a project: Web-Team (Web-Product-Roadmap).Nov 13 2024, 5:52 AM
NBaca-WMF added a parent task: T379707: [Hypothesis] [3.1.8] Content Discovery: Build Feature Additional Article Recommendations (Search A/B test and Article-recommendations A/B test).Nov 13 2024, 6:25 AM
NBaca-WMF edited projects, added Web-Team-Roadmap; removed Web-Team (Web-Product-Roadmap).Nov 13 2024, 7:05 AM
Jdlrobson moved this task from Epics/Goals to Groomed on the Web-Team-Backlog-Archived board.Nov 13 2024, 11:16 PM
Jdlrobson edited projects, added Web-Team; removed Web-Team-Roadmap, Web-Team-Backlog-Archived.Nov 13 2024, 11:35 PM
Jdlrobson moved this task from Incoming to Freezer on the Web-Team board.Nov 13 2024, 11:44 PM
NBaca-WMF moved this task from Freezer to Incoming on the Web-Team board.Nov 18 2024, 4:01 PM
NBaca-WMF edited projects, added Web-Team-Roadmap; removed Web-Team.Nov 18 2024, 7:29 PM
KSarabia-WMF subscribed.Nov 19 2024, 5:10 PM

Phabricator Summary: A/B Test Instrumentation Roadmap
Overview:
This roadmap outlines the steps required to instrument the mobile search A/B test for recommended articles, focusing on session length and user interactions. The process is broken into five phases:

  1. Planning and Scoping:
  • Confirm objectives and hypotheses.
  • Submit the session length mixin for legal/security review.
  1. Technical Enablers:
  • Finalize and modularize session length instrumentation.
  • Prepare the metrics schema.
  1. Drafting the Instrumentation Spec:
  • Define metrics (impressions, clicks, session data).
  • Address dynamic scenarios like type-ahead suggestions.
  1. Implementation: Log metrics and validate data for completeness.
  1. Analysis and Refinement:
  • Compare session length and interactions between test/control groups.
  • Iterate based on insights.

Next Steps:

  • Review the detailed roadmap for specific tasks, key questions, and data points.

Link to Full Roadmap

ovasileva added a subtask: T380596: Help with Legal and Security Review for New Mixin in Session Metrics Framework.Nov 22 2024, 4:39 PM
KSarabia-WMF added a subtask: T380630: [Spike] Investigate Stable User Identification for Logged-Out A/B Testing.Nov 22 2024, 4:39 PM
NBaca-WMF closed subtask T380630: [Spike] Investigate Stable User Identification for Logged-Out A/B Testing as Resolved.Nov 25 2024, 6:22 PM
Jdrewniak mentioned this in T378109: Mobile search A/B test instrumentation - technical enablers.Nov 26 2024, 2:17 PM
ovasileva closed subtask T378095: [Spike] Do people use the search within page button as Resolved.Nov 26 2024, 4:15 PM
Jdrewniak closed subtask T378101: [Spike] Decide where recommendations code should live as Resolved.Nov 26 2024, 9:56 PM
ovasileva closed subtask T378109: Mobile search A/B test instrumentation - technical enablers as Resolved.Dec 2 2024, 10:25 AM
ovasileva closed subtask T378100: [Spike] Functional + Design comparison of Typeahead search vs Mobile Search as Resolved.Dec 2 2024, 10:55 AM
Jdrewniak closed subtask T378103: Search recommendations A/B test: Write recommendations instrumentation spec as Resolved.Dec 5 2024, 7:02 PM
ovasileva added a subtask: T381604: Wire up new sessionLength instrumentation in search recommendations.Dec 9 2024, 8:55 AM
ovasileva added a subtask: T381616: Update Wikimedia Events to send experiment field to product_metrics schema.Dec 9 2024, 8:59 AM
ovasileva closed subtask T378118: [Spike] Check with Search Platform about traffic impact of recommendations A/B test as Resolved.Dec 9 2024, 11:00 AM
ovasileva closed subtask T378106: Decide on communication strategy for A/B test as Resolved.Dec 9 2024, 11:02 AM
ovasileva closed subtask T378115: Implement A/B test bucketing for mobile search recommendation as Resolved.Dec 17 2024, 4:00 PM
ovasileva closed subtask T380596: Help with Legal and Security Review for New Mixin in Session Metrics Framework as Resolved.
Jdlrobson closed subtask T378117: Search A/B test: Add MoreLike-based article suggestions when activating search bar in mobile as Resolved.Jan 2 2025, 5:52 PM
bwang closed subtask T381616: Update Wikimedia Events to send experiment field to product_metrics schema as Declined.Jan 10 2025, 4:12 PM
ovasileva added a subtask: T383266: Refine and document instrumentation spec and process with Jennifer.Jan 13 2025, 11:23 AM
ovasileva added a subtask: T382083: Modify how we bucket users to be based on localStorage rather than sessionStorage.
ovasileva added a subtask: T383422: Implement experimental sessionLength instrument for use in Search Recommendations A/B test.
ovasileva added a subtask: T380926: Configure new streams for the mobile search A/B test .
ovasileva added a subtask: T383358: Finish wiring up search recommendations A/B test click tracking.Jan 13 2025, 6:20 PM
ovasileva added a subtask: T383611: Create new click tracking instrumentation for search recomm A/B test.Jan 17 2025, 7:49 AM
SToyofuku-WMF changed the status of subtask T382083: Modify how we bucket users to be based on localStorage rather than sessionStorage from In Progress to Stalled.Jan 24 2025, 5:31 PM
Jdlrobson-WMF closed subtask T383266: Refine and document instrumentation spec and process with Jennifer as Resolved.Jan 24 2025, 10:18 PM
Jdlrobson-WMF closed subtask T383422: Implement experimental sessionLength instrument for use in Search Recommendations A/B test as Resolved.
Jdlrobson-WMF closed subtask T380926: Configure new streams for the mobile search A/B test as Resolved.
Jdlrobson-WMF closed subtask T383611: Create new click tracking instrumentation for search recomm A/B test as Resolved.
Jdlrobson changed the status of subtask T382083: Modify how we bucket users to be based on localStorage rather than sessionStorage from Stalled to Open.Jan 25 2025, 1:09 AM
Jdlrobson changed the status of subtask T382083: Modify how we bucket users to be based on localStorage rather than sessionStorage from Open to In Progress.
ovasileva added a subtask: T382080: Search recommendation clicks should trigger events.Jan 27 2025, 10:42 AM
Jdlrobson-WMF closed subtask T382080: Search recommendation clicks should trigger events as Resolved.Jan 27 2025, 7:34 PM
Jdlrobson-WMF closed subtask T381604: Wire up new sessionLength instrumentation in search recommendations as Resolved.Jan 27 2025, 8:22 PM
Jdlrobson-WMF closed subtask T383358: Finish wiring up search recommendations A/B test click tracking as Resolved.Jan 31 2025, 7:44 PM
Jdlrobson-WMF closed subtask T382083: Modify how we bucket users to be based on localStorage rather than sessionStorage as Resolved.Jan 31 2025, 10:39 PM
Jdlrobson-WMF added a subtask: T385309: Verify search recommendations A/B test click tracking is working as expected.Feb 7 2025, 6:42 PM
Jdlrobson-WMF added a subtask: T383936: Implement search activity id for Empty Search A/B test click tracking.
ovasileva added a subtask: T381345: [DESIGN] Auto-highlight the first recommendation in the empty state of search.Feb 10 2025, 9:06 AM
ovasileva added a subtask: T384019: Initial Empty Search A/B test Deployment.
Jdlrobson-WMF closed subtask T385309: Verify search recommendations A/B test click tracking is working as expected as Resolved.Feb 10 2025, 10:24 PM
Jdlrobson-WMF closed subtask T383936: Implement search activity id for Empty Search A/B test click tracking as Resolved.Feb 11 2025, 5:08 PM
Jdlrobson-WMF closed subtask T384019: Initial Empty Search A/B test Deployment as Resolved.Feb 19 2025, 5:24 PM
ovasileva added a subtask: T386735: Show empty search recommendation event is missing funnel data.Feb 25 2025, 9:22 AM
ovasileva added a subtask: T386495: Fix session tick mixin relating to when events fire.
ovasileva added a subtask: T386849: Deploy empty search AB test to all wikis.Feb 25 2025, 9:27 AM
jwang mentioned this in T387150: Data QA for instrumentation of empty search recommendation AB test.Feb 26 2025, 3:21 AM
bwang closed subtask T386735: Show empty search recommendation event is missing funnel data as Resolved.Feb 26 2025, 5:05 PM
bwang closed subtask T386495: Fix session tick mixin relating to when events fire as Resolved.
bwang changed the status of subtask T386849: Deploy empty search AB test to all wikis from Stalled to In Progress.Feb 26 2025, 5:24 PM
ovasileva added a subtask: T387400: SessionTick instrument should use sessionStorage instead of localStorage.Feb 27 2025, 8:42 AM
bwang changed the status of subtask T386849: Deploy empty search AB test to all wikis from In Progress to Stalled.Feb 27 2025, 3:49 PM
Jdlrobson-WMF changed the status of subtask T387400: SessionTick instrument should use sessionStorage instead of localStorage from Open to In Progress.Feb 27 2025, 5:50 PM
bwang closed subtask T387400: SessionTick instrument should use sessionStorage instead of localStorage as Resolved.Mar 4 2025, 11:36 PM
JScherer-WMF added subscribers: jwang, JScherer-WMF.Mar 18 2025, 5:18 PM

@jwang @ovasileva Do we have an analysis ticket for this yet?

ovasileva closed subtask T386849: Deploy empty search AB test to all wikis as Resolved.Mar 19 2025, 8:51 AM
ovasileva mentioned this in T392101: [Hypothesis] WE3.1.17 Search suggestions scaling.Apr 16 2025, 2:24 PM
bwang closed subtask T389326: Analyze results of search suggestions A/B test as Resolved.Apr 23 2025, 3:26 PM
Jdrewniak closed this task as Resolved.Aug 15 2025, 4:28 PM
Jdrewniak claimed this task.
Jdrewniak subscribed.

This work was completed in Q3 2024/25

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits