Page MenuHomePhabricator

No events being logged to product_metrics.web_base.search_ab_test_session_ticks
Closed, ResolvedPublic2 Estimated Story PointsBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Visit beta cluster
  • Enable event logging to developer console
mw.loader.using( 'mediawiki.api' ).then( function () {
  new mw.Api().saveOption( 'eventlogging-display-console', '1' );
} );
  • Refresh page
  • Confirm an event is fired to product_metrics.web_base.search_ab_test_session_ticks
  • Confirm after 30 seconds another event is fired to product_metrics.web_base.search_ab_test_session_ticks

What happens?:

What should have happened instead?:

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

Requirement

Verify that product_metrics.web_base.search_ab_test_session_ticks events are correctly logged in the beta cluster when the session starts and at 30-seconds.

BDD

Feature: Ensure product_metrics.web_base.search_ab_test_session_ticks events fire correctly  

  Scenario: Verify an event fires when the page loads  
    Given a user visits the beta cluster  
    When event logging is enabled in the developer console  
    And the page is refreshed  
    Then an event should be logged to product_metrics.web_base.search_ab_test_session_ticks  

  Scenario: Verify a second event fires after 30 seconds  
    Given an event was fired on page load  
    When 30 seconds pass  
    Then another event should be logged to product_metrics.web_base.search_ab_test_session_ticks

Test Steps

Test Case 1: Verify an event fires on page load

  1. Visit the beta cluster.
  2. Open the developer console.
  3. Enable event logging by running the following command:
mw.loader.using('mediawiki.api').then(function () {
  new mw.Api().saveOption('eventlogging-display-console', '1');
});
  1. Refresh the page.
  2. Observe the console and confirm that an event is logged to product_metrics.web_base.search_ab_test_session_ticks.
  3. AC1: An event fires on page load.

Test Case 2: Verify a second event fires after 30 seconds

  1. Follow Test Case 1 to enable event logging and refresh the page.
  2. Wait 30 seconds without interacting with the page.
  3. Observe the console and confirm that another event is logged to product_metrics.web_base.search_ab_test_session_ticks.
  4. AC2: A second event fires after 30 seconds.

Event Timeline

Change #1118885 had a related patch set uploaded (by Jdlrobson; author: Stoyofuku-wmf):

[mediawiki/extensions/WikimediaEvents@master] Fix session tick logging

https://gerrit.wikimedia.org/r/1118885

Change #1118885 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Fix session tick logging

https://gerrit.wikimedia.org/r/1118885

Should be on beta cluster soon - @Jdlrobson @Jdlrobson-WMF (not sure which one to use) do you want to coordinate with Edward to get this QAed in the next few hours?

Edtadros subscribed.

Status: ❌ FAIL
Environment: Beta
OS: macOS
Browser: Chrome
Device: MS

Test Case 1: Verify an event fires on page load

  1. Visit the beta cluster.
  2. Open the developer console.
  3. Enable event logging by running the following command:
mw.loader.using('mediawiki.api').then(function () {
  new mw.Api().saveOption('eventlogging-display-console', '1');
});
  1. Refresh the page.
  2. Observe the console and confirm that an event is logged to product_metrics.web_base.search_ab_test_session_ticks.
  3. ❌ AC1: An event fires on page load.

screenshot 8.png (945×1 px, 267 KB)

Test Case 2: Verify a second event fires after 30 seconds

  1. Follow Test Case 1 to enable event logging and refresh the page.
  2. Wait 30 seconds without interacting with the page.
  3. Observe the console and confirm that another event is logged to product_metrics.web_base.search_ab_test_session_ticks.
  4. ✅ AC2: A second event fires after 30 seconds.

screenshot 9.png (261×1 px, 68 KB)

status from meeting on search a/b test:

  • Session tick not firing 0 tick: The session tick should fire immediately one 0s, needs to be implemented?
  • We’re starting the regulator before we’re setting the mixin state.
  • Session length tick rate: The session tick is firing every 60s, not 30s, needs to be implemented.
  • Session tick reset: Sam confirmed we don’t need the reset, needs to be implemented

TODO:
@Jdrewniak to fix issue
@Edtadros to QA on beta cluster to confirm we are seeing events
@Jdrewniak to schedule deployment to production when testing is completed.

Change #1120641 had a related patch set uploaded (by Jdlrobson; author: Stoyofuku-wmf):

[mediawiki/extensions/WikimediaEvents@wmf/1.44.0-wmf.16] Fix session tick logging

https://gerrit.wikimedia.org/r/1120641

Change #1120641 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.44.0-wmf.16] Fix session tick logging

https://gerrit.wikimedia.org/r/1120641

Mentioned in SAL (#wikimedia-operations) [2025-02-18T22:20:08Z] <jdrewniak@deploy2002> Started scap sync-world: Backport for [[gerrit:1120641|Fix session tick logging (T386229)]]

Mentioned in SAL (#wikimedia-operations) [2025-02-18T22:23:06Z] <jdrewniak@deploy2002> jdrewniak, jdlrobson: Backport for [[gerrit:1120641|Fix session tick logging (T386229)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-02-18T22:31:21Z] <jdrewniak@deploy2002> Finished scap sync-world: Backport for [[gerrit:1120641|Fix session tick logging (T386229)]] (duration: 11m 13s)

Jdlrobson-WMF added subscribers: bwang, Jdrewniak.

Patch was backported to production. @bwang to close once we've confirmed we're seeing some events for section tick logging (likely in 4 hrs time)

We'll do data QA as part of next sprint in T386495

Bernard is having issues with superset, so I'll sign this off tomorrow morning using SELECT * FROM event.product_metrics_web_base_search_ab_test_session_ticks LIMIT 5

@Jdrewniak feel free to sign this off before I start my day if you are able to!

As of 10am ET, Feb 19 2025 - there are 130 rows in the event.product_metrics_web_base_search_ab_test_session_ticks table.