Background
In T376903, we discovered statsd/graphite could be used to cover the essential metrics we're seeking to obtain with our summarization experiment. This task covers wiring the metrics up so that the extension is sending data for the summary experiment
User story
- As an analyst, I want to be able to assess the success of the summarization experiment by looking at usage metrics
Requirements
- Count pageviews where summaries are displayed
- count summaries opened
- count clicks to yes
- count clicks to no
Acceptance criteria
The following metrics are sent to the statv endpoint
e.g: https://en.wikipedia.org/beacon/statsv?MediaWiki.experiment.web.summaries.enwiki.available=1c
- enabled - on every page load when the browser extension in enabled.
- available - on every page load when a simplified summary is available.
- opened - when a simplified summary is expanded.
- yes - when the 'was this helpful' yes button is clicked.
- no - when the 'was this helpful' no button is clicked.
BDD
Feature: Instrumentation for Summarization Experiment
Scenario: Send metrics for enabled state
Given the browser extension is installed and enabled
When the user visits a page
Then a metric is sent to the statsv endpoint with the label "enabled".
Scenario: Send metrics for available summaries
Given a simplified summary is available for a page
When the user visits the page
Then a metric is sent to the statsv endpoint with the label "available".
Scenario: Send metrics for opened summaries
Given a simplified summary is available on the page
When the user expands the summary
Then a metric is sent to the statsv endpoint with the label "opened".
Scenario: Send metrics for "Was this helpful?" feedback
Given a simplified summary is available on the page
When the user clicks the "yes" button
Then a metric is sent to the statsv endpoint with the label "yes".
And when the user clicks the "no" button
Then a metric is sent to the statsv endpoint with the label "no".Test Steps
Test Case 1: Verify Instrumentation for Summarization Metrics
- Ensure the browser extension is installed and enabled.
- Visit a page where summaries are enabled.
- AC1: Confirm a metric is sent to the statsv endpoint with the label “enabled”.
- AC2: Confirm a metric is sent to the statsv endpoint with the label “available” if a summary is present.
- Expand the summary on the page.
- AC3: Confirm a metric is sent to the statsv endpoint with the label “opened”.
- Interact with the “Was this helpful?” buttons:
- Click “yes”.
- AC4: Confirm a metric is sent to the statsv endpoint with the label “yes”.
- Click “no”.
- AC5: Confirm a metric is sent to the statsv endpoint with the label “no”.
QA Results - Prod
| AC | Status | Details |
|---|---|---|
| 1 | ✅ | T379214#10342774 |
| 2 | ✅ | T379214#10342774 |
| 3 | ✅ | T379214#10342774 |
| 4 | ✅ | T379214#10342774 |
| 5 | ✅ | T379214#10342774 |
Communication criteria - does this need an announcement or discussion?
- Discuss with @jwang and @ovasileva to ensure the metrics are sufficient for analysis. Discussion took place on https://phabricator.wikimedia.org/T374639
Rollback plan
Rollback should be fine - may need to document when if it would affect the data, but we should be okay