Background
We will be reusing the search satisfaction schema to perform two A/B tests on our changes to search:
- The first on the new location of the search widget.
- The second on the new widget / experience we're currently building in Vue.js.
These two A/B tests together support four possible configurations:
skinVersion | inputLocation | extraParams | Description |
legacy | "header-navigation" | Vector Legacy skin with Legacy search (current master) | |
latest | "header-navigation" | Vector Latest skin with Legacy search (current master) | |
latest | "header-moved" | Vector Latest skin with Legacy search and latest location | |
latest | "header-moved" | "WVUI" | Vector Latest skin with Latest search and latest location |
Both A/B tests will be executed in Vector's "Latest" (v2) mode only and compared to the Legacy mode (v1). In order to do this, we need to be able to distinguish between the old and new versions of the skin. As such, this task requires that a new skin version field be added. This field will only be needed for the search project and could be deleted later.
For the second test only, search can be configured within Vector's Latest mode to use the new experience (Vue.js) or the old experience. This will be set by wiki for at least logged-in users, although logged-in + anon is preferred. These will be distinguished in instrumentation by the addition of a second field for search version. We will also monitor the opt-in / out rate when new search is enabled and disabled.
Acceptance criteria
- Add a skin version field to /analytics/legacy/searchsatisfaction
- Add a search version field to /analytics/legacy/searchsatisfaction
Once the schema is updated (bumping it to 1.2.0, most likely), the version will also need to be updated in extension.json in WikimediaEvents:
"attributes": { "EventLogging": { "Schemas": { "SearchSatisfaction": "/analytics/legacy/searchsatisfaction/1.1.0",
Open/example patches
- The patch that introduces the required fields to the schema: https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/621473
- The (intentionally small) patch that sets the fields in the instrument: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/620012
Developer notes
- To access skin version reusing the logic in modules/ext.wikimediaEvents/desktopWebUIActions.js should work for now
- Ensure that our changes are isolated and can be disabled/reverted easily
- For completeness, Vector\Hooks::onOutputPageBodyAttributes should be updated to add a CSS class when $wgVectorUseCoreSearch is truthy. Otherwise we'll have to infer the presence of the Vue.js search widget implementation via something like
const isVueJSSearchWidgetPresent = !!document.querySelector( '#app .wvui-input' );
QA Results - Prod
AC | Status | Details |
---|---|---|
1 | ✅ | T256100#6529225 |
2 | ✅ | T256100#6529225 |
3 | ✅ | T256100#6529225 |
4 | ✅ | T256100#6529225 |