Page MenuHomePhabricator

[summaries] Instrument CTA modal
Closed, ResolvedPublic2 Estimated Story Points

Assigned To
Authored By
Jdrewniak
Mar 28 2025, 6:36 PM
Referenced Files
F60694814: screenshot 343.png
May 27 2025, 4:27 PM
F60694802: screenshot 342.png
May 27 2025, 4:27 PM
F60694796: screenshot 341.png
May 27 2025, 4:27 PM
F60694785: screenshot 340.png
May 27 2025, 4:27 PM
F58936582: Screenshot 2025-03-28 at 2.14.24 PM.png
Mar 28 2025, 6:36 PM
F58936579: Screenshot 2025-03-28 at 2.15.07 PM.png
Mar 28 2025, 6:36 PM
F58936577: Screenshot 2025-03-28 at 2.13.03 PM.png
Mar 28 2025, 6:36 PM
F58936575: Screenshot 2025-03-28 at 2.12.34 PM.png
Mar 28 2025, 6:36 PM

Description

Background

Assuming T387598 - Build CTA modal exists, instrumentation needs to be added to the modal to enable data collection on how many people choose to enable vs disable the experiment.

User story

As a product team, we need to know how many people opt-in to the summaries experiment.

Requirements

Add instrumentation that logs the following 4 events when the CTA is open.

screenshot
Screenshot 2025-03-28 at 2.12.34 PM.png (658×1,194 px, 244 KB)
Screenshot 2025-03-28 at 2.13.03 PM.png (694×1,084 px, 221 KB)
Screenshot 2025-03-28 at 2.15.07 PM.png (622×1,048 px, 227 KB)
Screenshot 2025-03-28 at 2.14.24 PM.png (614×1,058 px, 222 KB)
descriptionCTA is shown'more info' link click'no' or dismiss click'yes' click
actionshowclickclickclick
action_subtypeshow_summary_onboardinglearn_morenoyes
action_sourcesummary_onboardingsummary_onboardingsummary_onboardingsummary_onboarding

Since the CTA is meant to be shown only as a centralNotice banner, it should be able to invoke it programatically via a js hook in the browser console.

note: If the modal can be dismissed by clicking outside of it, that should be tracked as a 'no' click.

BDD

Feature: Instrumentation for article summary interaction

Scenario: Summary link is available on page
  Given I visit a mobile article page with a simple summary available
  When the summary banner renders
  Then an event with action "show" action_subtype "show_summary_onboarding" and action_source "summary_onboarding " is fired

Scenario: User clicks "Learn More"
  Given I see a summary banner with a "Learn More" link
  When I click the "Learn More" link
  Then an event with action "click" action_subtype "learn_more" and action_source "summary_onboarding " is fired

Scenario: User clicks "No, thanks" feedback on summary
  Given the summary overlay is visible
  When I click the "No, thanks" button
Then an event with action "click" action_subtype "no" and action_source "summary_onboarding " is fired

Scenario: User clicks "Enable" feedback on summary
  Given the summary overlay is visible
  When I click the "Enable" button
Then an event with action "click" action_subtype "yes" and action_source "summary_onboarding " is fired

Test Steps

Test Case 1: Summary link is available on page

  1. https://en.m.wikipedia.beta.wmflabs.org/wiki/Paris as an anonymous user.
  2. Confirm the ArticleSummaries feature is enabled and the summary banner renders at the top.
  3. Open browser DevTools → Console.
  4. Run:
mw.hook('ext.articleSummaries.cta.open').fire();
  1. Reload the page.
  2. AC1: Confirm the console logs an event with:

• action: "show"
• action_subtype: "show_summary_onboarding"
• action_source: "summary_onboarding"

Test Case 2: User clicks “Learn More”

  1. With the summary overlay visible (via banner or hook).
  2. Click the “Learn More” link.
  3. AC2: Confirm the console logs an event with:

• action: "click"
• action_subtype: "learn_more"
• action_source: "summary_onboarding"

Test Case 3: User clicks “No, thanks” feedback on summary

  1. With the summary overlay visible (via banner or hook)
  2. In the overlay, click the “No, thanks” link..
  3. AC3: Confirm the console logs an event with:

• action: "click"
• action_subtype: "no"
• action_source: "summary_onboarding"

Test Case 4: User clicks “Enable” feedback on summary

  1. With the summary overlay visible (via banner or hook)
  2. In the overlay, click the “Enable” link..
  3. AC4: Confirm the console logs an event with:

• action: "click"
• action_subtype: "yes"
• action_source: "summary_onboarding"

QA Results - Beta

Acceptance criteria

  • The CTA is instrumented an fires the events with the data described above.

This task was created by Version 1.2.0 of the Web team task template using phabulous

Event Timeline

Jdrewniak triaged this task as High priority.
Jdlrobson-WMF changed the task status from Open to In Progress.Mar 31 2025, 5:12 PM
Jdlrobson-WMF moved this task from Incoming to Q4 on the Web-Team board.
Jdlrobson-WMF subscribed.

Ideally we should get this ready for sprint 1.

Jdlrobson-WMF set the point value for this task to 3.
Jdlrobson-WMF moved this task from Q4 to Sprint Backlog on the Web-Team board.
Jdlrobson-WMF changed the task status from In Progress to Open.Apr 23 2025, 7:14 AM
LMora-WMF lowered the priority of this task from High to Medium.Apr 23 2025, 5:52 PM

Started working on this since there wasn't anything on the sprint board in 'Ready for development'

Thanks for not moving it onto the board! (we can consider that later in the sprint when the other stuff is taken care of!)

Change #1146651 had a related patch set uploaded (by LorenMora; author: LorenMora):

[mediawiki/extensions/WikimediaEvents@master] Instrument CTA modal

https://gerrit.wikimedia.org/r/1146651

Change #1146653 had a related patch set uploaded (by LorenMora; author: LorenMora):

[mediawiki/extensions/ArticleSummaries@master] Instrument CTA modal

https://gerrit.wikimedia.org/r/1146653

Jdlrobson-WMF raised the priority of this task from Medium to High.May 15 2025, 5:45 PM

I submitted two patches for this and I believe that's all the work that's needed. I tested everything locally and it's working as expected. All tests are passing except there is a Sonar warning for WikimediaEvents. Next steps would be for someone to review this and if you could look into the Sonar warning. Thank you!

Jdlrobson-WMF changed the point value for this task from 3 to 2.May 21 2025, 5:52 PM
Jdlrobson-WMF updated Other Assignee, added: Jdlrobson-WMF.

We will take a look at this since we are already looking at instrumentation.

Change #1146651 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Instrument CTA modal

https://gerrit.wikimedia.org/r/1146651

Change #1146653 merged by jenkins-bot:

[mediawiki/extensions/ArticleSummaries@master] Instrument CTA modal

https://gerrit.wikimedia.org/r/1146653

@Edtadros you should be able to test this in the beta cluster just how we tested T387406: [summaries] Create instrument + wire-up summary UI let Kim know if you have any questions as she's the best placed to answer! Thanks for reviewing Kim!

Jdlrobson-WMF lowered the priority of this task from High to Medium.May 23 2025, 10:14 PM
Edtadros subscribed.

Test Steps

Status: ✅ PASS
Environment: Beta
OS: macOS
Browser: Chrome
Device: MS

Test Case 1: Summary link is available on page

  1. https://en.m.wikipedia.beta.wmflabs.org/wiki/Paris as an anonymous user.
  2. Confirm the ArticleSummaries feature is enabled and the summary banner renders at the top.
  3. Open browser DevTools → Console.
  4. Run:
mw.hook('ext.articleSummaries.cta.open').fire();
  1. Reload the page.
  2. ✅ AC1: Confirm the console logs an event with:

• action: "show"
• action_subtype: "show_summary_onboarding"
• action_source: "summary_onboarding"

screenshot 340.png (1,397×1,453 px, 449 KB)

Test Case 2: User clicks “Learn More”

  1. With the summary overlay visible (via banner or hook).
  2. Click the “Learn More” link.
  3. ✅ AC2: Confirm the console logs an event with:

• action: "click"
• action_subtype: "learn_more"
• action_source: "summary_onboarding"

screenshot 341.png (1,397×1,453 px, 450 KB)

Test Case 3: User clicks “No, thanks” feedback on summary

  1. With the summary overlay visible (via banner or hook)
  2. In the overlay, click the “No, thanks” link..
  3. ✅ AC3: Confirm the console logs an event with:

• action: "click"
• action_subtype: "no"
• action_source: "summary_onboarding"

screenshot 342.png (1,397×1,453 px, 461 KB)

Test Case 4: User clicks “Enable” feedback on summary

  1. With the summary overlay visible (via banner or hook)
  2. In the overlay, click the “Enable” link..
  3. ✅ AC4: Confirm the console logs an event with:

• action: "click"
• action_subtype: "yes"
• action_source: "summary_onboarding"

screenshot 343.png (1,397×1,453 px, 454 KB)

Just checked this locally and on beta cluster and was also able to verify that events are being sent.