Follow up to T383358
Background
In T383358 we hit a few issues QAing that the instrumentation was working as expected due to configuration on the beta cluster.
User story
As a web team member I want to launch an A/B test that collects the information we need.
Requirements
- We have configured the instrumentation to work correctly
- We have run the QA steps in T383358
BDD
Feature: Search Recommendations Click Tracking
Scenario: Track search session initiation
Given a user is on a Wikimedia search page
When the empty search box appears
Then an event with action 'init' and action_subtype 'init_search_box' should be fired
Scenario: Track clicking on the empty search box
Given a user is on a Wikimedia search page
When the user clicks inside the empty search box
Then an event with action 'click' and action_source 'search_box' should be fired
Scenario: Track impressions of empty-state recommendations
Given a user is on a Wikimedia search page
When the list of empty-state recommendations appears
Then an event with action 'show' and action_subtype 'show_empty_state_recommendation' should be fired
Scenario: Track when user starts typing
Given a user is on a Wikimedia search page
When the user types the first letter into the search box
Then an event with action 'type' and action_source 'search_box' should be fired
Scenario: Track impressions of type-ahead search recommendations
Given a user is on a Wikimedia search page
When type-ahead recommendations appear
Then an event with action 'show' and action_subtype 'show_autocomplete_recommendation' should be fired
Scenario: Track clicks on empty-state recommendations
Given a user is on a Wikimedia search page
When the user clicks on an empty-state recommendation
Then an event with action 'click' and action_source 'empty_search_suggestion' should be fired
Scenario: Track clicks on type-ahead recommendations
Given a user is on a Wikimedia search page
When the user clicks on a type-ahead recommendation
Then an event with action 'click' and action_source 'autocomplete_search_suggestion' should be firedTest Steps
Test Case 1: Verify event fires for search session initiation
- Open an incognito window and visit https://en.m.wikipedia.beta.wmflabs.org/w/index.php?title=T352930.
- Log in to a user account.
- Open the JavaScript console and execute:
mw.loader.using('mediawiki.api').then(function () {
new mw.Api().saveOption('eventlogging-display-console', '1');
});- Ensure the search bar is visible.
- Click on the empty search box.
- Verify that an event fires with:
• Event Name: product_metrics.web_base.search_ab_test_clicks
• Action: init
• Action Subtype: init_search_box
- AC1: Event fires when the empty search box appears.
Test Case 2: Verify event fires when user clicks on the empty search box
- Follow Test Case 1 steps until step 4.
- Click inside the search box.
- Verify that an event fires with:
• Event Name: product_metrics.web_base.search_ab_test_clicks
• Action: click
• Action Source: search_box
- AC2: Event fires when the user clicks on the empty search box.
Test Case 3: Verify event fires when empty-state recommendations appear
- Follow Test Case 1 steps until step 4.
- Ensure that the empty-state recommendations appear in the search box dropdown.
- Verify that an event fires with:
• Event Name: product_metrics.web_base.search_ab_test_clicks
• Action: show
• Action Subtype: show_empty_state_recommendation
- AC3: Event fires when empty-state recommendations appear.
Test Case 4: Verify event fires when user starts typing the first letter
- Follow Test Case 1 steps until step 4.
- Type a single letter in the search box.
- Verify that an event fires with:
• Event Name: product_metrics.web_base.search_ab_test_clicks
• Action: type
• Action Source: search_box
- AC4: Event fires when user types the first letter.
Test Case 5: Verify event fires when type-ahead recommendations appear
- Follow Test Case 4 steps until step 2.
- Ensure that type-ahead recommendations appear in the search box dropdown.
- Verify that an event fires with:
• Event Name: product_metrics.web_base.search_ab_test_clicks
• Action: show
• Action Subtype: show_autocomplete_recommendation
- AC5: Event fires when type-ahead recommendations appear.
Test Case 6: Verify event fires when user clicks on an empty-state recommendation
- Follow Test Case 3 steps until step 2.
- Click on one of the empty-state recommendations.
- Verify that an event fires with:
• Event Name: product_metrics.web_base.search_ab_test_clicks
• Action: click
• Action Source: empty_search_suggestion
- AC6: Event fires when user clicks on an empty-state recommendation.
Test Case 7: Verify event fires when user clicks on a type-ahead recommendation
- Follow Test Case 5 steps until step 2.
- Click on one of the type-ahead recommendations.
- Verify that an event fires with:
• Event Name: product_metrics.web_base.search_ab_test_clicks
• Action: click
• Action Source: autocomplete_search_suggestion
- AC7: Event fires when user clicks on a type-ahead recommendation.
Design
- Add mockups and design requirements
Acceptance criteria
- If any issues are discovered that block deploying A/B test, tasks are created, estimated and the sprint is revised to accomodate them.
Communication criteria - does this need an announcement or discussion?
- Add communication criteria
Rollback plan
- What is the rollback plan in production for this task if something goes wrong?
QA Results - Beta
| AC | Status | Details |
|---|---|---|
| 1 | ✅ | T385309#10533374 |
| 2 | ✅ | T385309#10533374 |
| 3 | ✅ | T385309#10533374 |
| 4 | ✅ | T385309#10533374 |
| 5 | ✅ | T385309#10533374 |
| 6 | ✅ | T385309#10533374 |
| 7 | ✅ | T385309#10533374 |
This task was created by Version 1.2.0 of the Web team task template using phabulous