Page MenuHomePhabricator

Show empty search recommendation event is missing funnel data
Closed, ResolvedPublic3 Estimated Story Points

Description

Background

A bugs was discovered during Jennifer's QA of eu/ca events
https://phabricator.wikimedia.org/T386142#10553847

  • Show empty search recommendation event is missing funnel data

Requirements

  • Show empty search recommendation event has funnel data with the search activity id
  • fix(es) are backported and QAed in production

Acceptance criteria

  • Show empty search recommendations should have a valid funnel_entry_token
SELECT dt, action, action_context, action_source, action_subtype, funnel_entry_token
FROM event.product_metrics_web_base_search_ab_test_clicks
WHERE year=2025 and month=2
AND performer.session_id='0503d55281c5066deefa'
ORDER BY dt
limit 100

Developer notes

I can reproduce Show empty search recommendation event is missing funnel data. Looks like ext.MobileFrontend.searchOverlay.empty hook fires before ext.MobileFrontend.searchOverlay.open as the constructor calls resetSearch
We'd need to either:
Adjust MobileFrontend code so the hook firing order is as expected
In WikimediaEvents update the hook callback for ext.MobileFrontend.searchOverlay.empty so that the event is delayed until it's available (e.g. via a setInterval).

BDD

Feature: Ensure Show Empty Search Recommendation event includes funnel data  

  Scenario: Verify Show Empty Search Recommendation event contains search activity ID  
    Given a user activates the search box  
    When the empty search recommendations are displayed  
    Then the event should include a valid funnel_entry_token

Test Steps

Test Case 1: Verify Show Empty Search Recommendation event contains funnel data

  1. Open the developer console.
  2. Enable event logging by running:
mw.loader.using('mediawiki.api').then(function () {
  new mw.Api().saveOption('eventlogging-display-console', '1');
});
  1. Activate the search box without entering a query.
  2. Observe the console logs for the Show Empty Search Recommendation event.
  3. Confirm that the event includes a valid funnel_entry_token with the search activity ID.
  4. AC1: The event includes a valid funnel_entry_token.

QA Results - Beta

ACStatusDetails
1T386735#10577109

Event Timeline

bwang triaged this task as High priority.Feb 18 2025, 6:24 PM
Jdrewniak changed the task status from Open to In Progress.Feb 18 2025, 6:56 PM
Jdrewniak assigned this task to bwang.
Jdrewniak moved this task from Incoming to Sprint Backlog on the Web-Team board.
Jdrewniak moved this task from Sprint Backlog to Incoming on the Web-Team board.
Jdrewniak moved this task from Incoming to Q3 on the Web-Team board.
Jdlrobson-WMF renamed this task from Fix remaining Search AB test bugs to Show empty search recommendation event is missing funnel data.Feb 19 2025, 4:56 PM
Jdlrobson-WMF updated the task description. (Show Details)
Jdlrobson-WMF subscribed.

I descoped this to be about the funnel data bug. I am seeing click tracking events. I suspect if we increase the sampling rate per plan in T386734 we will see the click events.

Jdlrobson-WMF set the point value for this task to 3.Feb 19 2025, 5:33 PM
Jdlrobson-WMF moved this task from Q3 to Sprint Backlog on the Web-Team board.

Change #1121699 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[mediawiki/extensions/MobileFrontend@master] Update ext.MobileFrontend.searchOverlay.empty hook to fire after ext.MobileFrontend.searchOverlay.open

https://gerrit.wikimedia.org/r/1121699

Change #1121699 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Update ext.MobileFrontend.searchOverlay.empty hook to fire after ext.MobileFrontend.searchOverlay.open

https://gerrit.wikimedia.org/r/1121699

Edtadros subscribed.

Status: ✅ PASS
Environment: Beta
OS: macOS
Browser: Chrome
Device: MS

Test Case 1: Verify Show Empty Search Recommendation event contains funnel data

  1. Open the developer console.
  2. Enable event logging by running:
mw.loader.using('mediawiki.api').then(function () {
  new mw.Api().saveOption('eventlogging-display-console', '1');
});
  1. Activate the search box without entering a query.
  2. Observe the console logs for the Show Empty Search Recommendation event.
  3. Confirm that the event includes a valid funnel_entry_token with the search activity ID.
  4. ✅ AC1: The event includes a valid funnel_entry_token.

It's not entirely clear, but I suspect what you are hoping to see is that for the action_subtype: 'show_empty_state_recommendation' you also get a funnel_entry_token. Below are the three events that fire once clicked into search but before typing anything. The second expanded action looks to be what satisfies this task.

screenshot 45.png (1×1 px, 326 KB)

Change #1122254 had a related patch set uploaded (by Jdlrobson; author: Bernard Wang):

[mediawiki/extensions/MobileFrontend@wmf/1.44.0-wmf.17] Update ext.MobileFrontend.searchOverlay.empty hook to fire after ext.MobileFrontend.searchOverlay.open

https://gerrit.wikimedia.org/r/1122254

Jdlrobson-WMF added a subscriber: SToyofuku-WMF.

@SToyofuku-WMF can I assign to you for sign off (as we should backport https://gerrit.wikimedia.org/r/1122254 before resolving it?)

Change #1122254 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@wmf/1.44.0-wmf.17] Update ext.MobileFrontend.searchOverlay.empty hook to fire after ext.MobileFrontend.searchOverlay.open

https://gerrit.wikimedia.org/r/1122254

Mentioned in SAL (#wikimedia-operations) [2025-02-25T22:28:52Z] <toyofuku@deploy2002> Started scap sync-world: Backport for [[gerrit:1122254|Update ext.MobileFrontend.searchOverlay.empty hook to fire after ext.MobileFrontend.searchOverlay.open (T386735)]], [[gerrit:1122633|Deploy Search AB test to french wiki]]

Mentioned in SAL (#wikimedia-operations) [2025-02-25T22:31:50Z] <toyofuku@deploy2002> bwang, toyofuku, jdlrobson: Backport for [[gerrit:1122254|Update ext.MobileFrontend.searchOverlay.empty hook to fire after ext.MobileFrontend.searchOverlay.open (T386735)]], [[gerrit:1122633|Deploy Search AB test to french wiki]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-02-25T22:52:19Z] <toyofuku@deploy2002> Finished scap sync-world: Backport for [[gerrit:1122254|Update ext.MobileFrontend.searchOverlay.empty hook to fire after ext.MobileFrontend.searchOverlay.open (T386735)]], [[gerrit:1122633|Deploy Search AB test to french wiki]] (duration: 23m 26s)

Status: ✅ PASS
Environment: frwiki
OS: macOS
Browser: Chrome
Device: MS

Test Case 1: Verify Show Empty Search Recommendation event contains funnel data

  1. Open the developer console.
  2. Enable event logging by running:
mw.loader.using('mediawiki.api').then(function () {
  new mw.Api().saveOption('eventlogging-display-console', '1');
});
  1. Activate the search box without entering a query.
  2. Observe the console logs for the Show Empty Search Recommendation event.
  3. Confirm that the event includes a valid funnel_entry_token with the search activity ID.
  4. ✅ AC1: The event includes a valid funnel_entry_token.

screenshot 46.png (1×1 px, 298 KB)