Background
If we introduce a pre-generated summary feature as an opt-in feature on a the mobile site of a production wiki, we will be able to measure a CTR greater than 4%, ensure no negative effects to session length, pageviews, or internal referrals, and use this data to decide how and if we will further scale the summary feature."
We want to know if user's find this feature useful or not.
User story
As a reader I want to tell WMF what I think of their new feature so that my voice is heard!
Requirements
Per the instrumentation spec for this experiment, the following *5 event* are logged.
 |  |  |  |  |
| User lands on the page containing the summary link. | User clicks on the Read summary link | The ‘Summary’ overlay shows | User clicks on 'Yes' to the question "Was this summary helpful" | User clicks on 'No' to the question "Was this summary helpful" |
| summary_available | summary_activated | summary_shown | summary_approved | summary_disapproved |
BDD
Feature: Instrumentation for article summary interaction Scenario: Summary link is available on page Given I visit a mobile article page with a simple summary available When the summary banner renders Then an event of type "summary_available" is fired Scenario: User clicks "Read summary" Given I see a summary banner with a "Read summary" link When I click the "Read summary" link Then an event of type "summary_activated" is fired Scenario: Summary overlay is shown Given I have clicked the "Read summary" link When the overlay becomes visible Then an event of type "summary_shown" is fired Scenario: User clicks "Yes" feedback on summary Given the summary overlay is visible When I click the "Yes" button Then an event of type "summary_approved" is fired Scenario: User clicks "No" feedback on summary Given the summary overlay is visible When I click the "No" button Then an event of type "summary_disapproved" is fired
Test Steps
Test Case 1: Verify summary_available event
- Logged out: Navigate to a mobile article page where a summary is available e.g. https://en.m.wikipedia.beta.wmflabs.org/wiki/Paris
- Ensure that the summary banner loads successfully.
- Open the network tab, switch tab back and forth and verify eventlogging events fired.
- AC1: Confirm that a summary_available event is logged.
Test Case 2: Verify summary_activated event
- Click on the “Read summary” link in the banner.
- AC2: Confirm that a summary_activated event is logged.
Test Case 3: Verify summary_shown event
- Observe the summary overlay appearing after clicking “Read summary”.
- AC3: Confirm that a summary_shown event is logged.
Test Case 4: Verify summary_approved event
- In the summary overlay, click the “Yes” button to answer “Was this summary helpful?”.
- AC4: Confirm that a summary_approved event is logged.
Test Case 5: Verify summary_disapproved event
- Refresh the page or load another article with a summary.
- Click “Read summary” to bring up the overlay.
- Click the “No” button.
- AC5: Confirm that a summary_disapproved event is logged.
QA Results - Beta
| AC | Status | Details |
|---|---|---|
| 1 | ✅ | T387406#10842222 |
| 2 | ✅ | T387406#10842222 |
| 3 | ✅ | T387406#10842222 |
| 4 | ✅ | T387406#10842222 |
| 5 | ✅ | T387406#10842222 |
Acceptance criteria
- The events are logged to the new stream in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1129958
- The stream in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1129958 is deployed to production when we are ready. For context this was implemented in T389097: [summaries] Create new data collection stream for summaries mobile pilot
Communication criteria - does this need an announcement or discussion?
- Add communication criteria
Rollback plan
Code is not in production so no rollback plan needed.
This task was created by Version 1.2.0 of the Web team task template using phabulous
