Page MenuHomePhabricator

Search A/B test: Add MoreLike-based article suggestions when activating search bar in mobile
Closed, ResolvedPublic3 Estimated Story Points

Description

Background

  • We are testing content recommendations on mobile. This task is to add those recommendations to the search bar when activated

User story

  • As a user in the appropriate test group, I see article recommendations when I activate the search bar, so I can potentially deepen my relationship with Wiki and more easily fall into "rabbit holes"

Requirements

  • Must work on mobile (if it works on desktop this is a positive side effect)
  • Must utilize the MoreLike API currently in use for Related Articles to generate three recommendations
  • Must be disabled by default
  • When the experiment is enabled 50% of the sample group see the recommendations, 50% of the sample group. get the existing behaviour
  • WIkimediaEvents is able to determine if the experiment is running or not.

BDD

Feature: Display MoreLike-based article suggestions in mobile search bar
  Scenario: Article recommendations appear on activating the search bar
    Given the user is on the mobile site
    When the user activates the search bar
    Then MoreLike-based article suggestions should be displayed below the search bar

Test Steps

Test Case 1: Validate recommendations display when activating the search bar

  1. Navigate to the mobile site.
  2. Ensure the search bar is visible on the page.
  3. Activate the search bar by tapping on it.
  4. Verify that MoreLike-based article suggestions appear below the search bar.
  5. AC1: Recommendations are displayed when the search bar is activated.

Design

  • More detailed designs to come

Acceptance criteria

  • We have added 3 MoreLike-based search results to treatment group users on mobile.
  • These results do not appear for the control group
  • We have tied this into instrumentation as part of https://phabricator.wikimedia.org/T378109

The solution should be minimal and comprise of the following components based on our discussion in T378109: Mobile search A/B test instrumentation - technical enablers:

Communication criteria - does this need an announcement or discussion?

  • Yes - we need to document and communicate how this works and our reasoning for this experiment

Rollback plan

  • What is the rollback plan in production for this task if something goes wrong?

This work needs to only apply to the treatment group

This task was created by Version 1.2.0 of the Web team task template using phabulous

QA Results - Beta

ACStatusDetails
1T378117#10399296

Details

Event Timeline

ovasileva renamed this task from Add MoreLike-based article suggestions when activating search bar in mobile to Search A/B test: Add MoreLike-based article suggestions when activating search bar in mobile.Nov 7 2024, 8:44 AM
ovasileva triaged this task as High priority.
ovasileva updated the task description. (Show Details)

Discussion to follow this week to further refine this ticket based on designs / follow up on https://phabricator.wikimedia.org/T378100 - based on the outcome of this, we should create a relevant spec and estimate between now and next sprint

Based off T378100, the decision was to use RelatedArticles inside the existing mobile search experience.

@Jdlrobson put together a helpful MVP here: https://patchdemo.wmcloud.org/wikis/a5187de38d/w/index.php?title=Related&mobileaction=toggle_view_mobile#/search
This POC is accomplished with the following 2 patches
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/RelatedArticles/+/1092871?usp=search
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/1092870?usp=search

The design would be similar to this, but more refined.

Screenshot 2024-11-20 at 11.10.13 AM.png (2,048×822 px, 156 KB)

@bwang will you faciliate an estimation for this so we can get it queued up for next sprint?

Jdlrobson updated the task description. (Show Details)
Jdlrobson added a subscriber: JScherer-WMF.

@JScherer-WMF please add design to this one when you've had a chance to adapt the design based on the proof of concept at https://patchdemo.wmcloud.org/wikis/a5187de38d/w/index.php?title=Related#/search

Jdlrobson renamed this task from Search A/B test: Add MoreLike-based article suggestions when activating search bar in mobile to Search A/B test: Add MoreLike-based article suggestions to RelatedArticles when activating search bar in mobile.Nov 20 2024, 10:38 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson added a project: RelatedArticles.
bwang renamed this task from Search A/B test: Add MoreLike-based article suggestions to RelatedArticles when activating search bar in mobile to Search A/B test: Add MoreLike-based article suggestions when activating search bar in mobile.Nov 21 2024, 4:55 PM
bwang updated the task description. (Show Details)
JScherer-WMF moved this task from Incoming to Sprint Backlog on the Web-Team board.
JScherer-WMF added a subscriber: Jdlrobson-WMF.

Looks good @Jdlrobson-WMF except that read-more-container-heading needs padding: 12px; and read-more-container-card-list needs padding: 12px; as well.

End result should look like this:

Screenshot 2024-11-22 at 2.53.01 PM.png (752×1,346 px, 129 KB)

Let me know if you need any other design input on this one.

bwang set the point value for this task to 2.Nov 25 2024, 5:44 PM
bwang changed the point value for this task from 2 to 3.Nov 26 2024, 6:16 PM
SToyofuku-WMF subscribed.

can be done in parallel with T378115, but worth coordinating to make sure we understand how to gate effectively

Change #1101601 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/RelatedArticles@master] Improve styling of empty search experiment

https://gerrit.wikimedia.org/r/1101601

Change #1101601 merged by jenkins-bot:

[mediawiki/extensions/RelatedArticles@master] Improve styling of empty search experiment

https://gerrit.wikimedia.org/r/1101601

@JScherer-WMF please review the design is sufficient for an A/B test. I optimized the design for mobile. Note it was near impossible to style it correctly on desktop/tablet without significant improvements to the codebase so I just did the best I ctould there. Hope that's okay. We can ignore users with these kind of viewports in the A/B test if we think that's significant to disrupt.

Screenshot 2024-12-10 at 11.43.44 AM.png (738×502 px, 42 KB)

https://en.m.wikipedia.beta.wmflabs.org/wiki/Polar_bear#/search

Edtadros subscribed.

Status: ✅ PASS
Environment: Beta
OS: macOS
Browser: Chrome
Device: MS

Test Case 1: Validate recommendations display when activating the search bar

  1. Navigate to the mobile site.
  2. Ensure the search bar is visible on the page.
  3. Activate the search bar by tapping on it.
  4. Verify that MoreLike-based article suggestions appear below the search bar.
  5. AC1: Recommendations are displayed when the search bar is activated.

screenshot 53.mov.gif (1,184×1,016 px, 383 KB)

ovasileva subscribed.

@JScherer-WMF - just double-checking that this has passed design review?

Yes. I reviewed this with Justin.