This (currently skeletal) task covers the work to define what's important to measure in the new but not the old search experience. Try to reuse Schema:SearchSatisfaction (YAML, instrument). It was recently ported to the Modern Event Platform.
Requirements identified
Instrumentation requirements and => mapping to the SearchSatisfaction schema:
- Selecting the search widget => Not instrumented. An event is not sent until a search session begins when a user starts typing in the search widget. This either needs to be added to the schema or removed from the requirements!
- Begin of search session (user enters text) =>action: "searchResultPage", source: "autocomplete" is recorded everytime a user begins to enter text into the search widget.
- End of search session (user selects a search result)=> action: "click"
- End of search session (user closes search widget) => There are multiple ways this can be determined using SearchSatisfaction. Finding all search sessions that do not include a click action will show users that ended a session before selecting a search result. Also, if no search is performed within ten minutes of a previous search a new searchSessionid is generated. There is also a mwSessionId which should end when the browser has fully closed.
- [Nice to have] End of search session due to submitting the form (i.e. not clicking on an autocomplete result) => action: click, position: >= -1 followed by action: searchResultPage
Additional requirements:
- We will be performing an A/B test of the new treatment for logged-in users for at least one of our test wikis. That's covered by T251740.
Acceptance criteria
- T257698 is revised as needed.
- The Discovery-ARCHIVED team (see T130321) is pinged to ensure we do not interfere with their metrics.
- No requirements around where the code lives but historically that's been MediaWiki-extensions-WikimediaEvents.
References
- Portal search box assessment
- Schema:Search
- Schema:MobileWebSearch (client implementation)
- Schema:SearchSatisfaction
- Performance team meeting minutes for T250336
- MachineVision logger plugin implementation
Open questions
- Do we need any X-Analytics headers for skin version (e.g., 1 (or not present) = Legacy, 2 = Latest) or search version (e.g., search=1 and search=2)? It's possible that a test wiki may have the latest version of the skin deployed but the new Vue.js search disabled) at least initially. The new searches should be using the new MediaWiki REST API.