Page MenuHomePhabricator

[Spike] What should we instrument in the new Vue.js search experience?
Closed, ResolvedPublicSpike

Description

This (currently skeletal) task covers the work to define what's important to measure in the new but not the old search experience. Try to reuse Schema:SearchSatisfaction (YAML, instrument). It was recently ported to the Modern Event Platform.

Requirements identified

Instrumentation requirements and => mapping to the SearchSatisfaction schema:

  • Selecting the search widget => Not instrumented. An event is not sent until a search session begins when a user starts typing in the search widget. This either needs to be added to the schema or removed from the requirements!
  • Begin of search session (user enters text) =>action: "searchResultPage", source: "autocomplete" is recorded everytime a user begins to enter text into the search widget.
  • End of search session (user selects a search result)=> action: "click"
  • End of search session (user closes search widget) => There are multiple ways this can be determined using SearchSatisfaction. Finding all search sessions that do not include a click action will show users that ended a session before selecting a search result. Also, if no search is performed within ten minutes of a previous search a new searchSessionid is generated. There is also a mwSessionId which should end when the browser has fully closed.
  • [Nice to have] End of search session due to submitting the form (i.e. not clicking on an autocomplete result) => action: click, position: >= -1 followed by action: searchResultPage

Additional requirements:

  • We will be performing an A/B test of the new treatment for logged-in users for at least one of our test wikis. That's covered by T251740.

Acceptance criteria

References

Open questions

  • Do we need any X-Analytics headers for skin version (e.g., 1 (or not present) = Legacy, 2 = Latest) or search version (e.g., search=1 and search=2)? It's possible that a test wiki may have the latest version of the skin deployed but the new Vue.js search disabled) at least initially. The new searches should be using the new MediaWiki REST API.

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptApr 3 2020, 7:12 PM
Niedzielski renamed this task from [Spike] What should we instrument in the new search experience? to [Spike] What should we instrument in the new Vue.js search experience?.Apr 5 2020, 4:20 PM

@Mayakp.wiki, @MNeisler, will Schema:SearchSatisfaction with a skin version field (T256100) satisfy the instrumentation requirements for Vue.js search? Note: the A/B test was covered in T251740.

@Mayakp.wiki, @MNeisler, will Schema:SearchSatisfaction with a skin version field (T256100) satisfy the instrumentation requirements for Vue.js search? Note: the A/B test was covered in T251740.

@Niedzielski I reviewed the SearchSatisfaction schema and confirmed it satisfies all the instrumentation requirements as identified above with the exception of Selecting the search widget. Based on my review, it looks like an event is not recorded in SearchSatisfaction until a user starts typing in the search widget, which marks the start of a search session.
Do we need to track when someone clicks on the search widget without typing? If so, we will need this added in addition to the skin field.

Here is a mapping of metrics within that schema to the requirements above:

Selecting the search widget => Not instrumented. An event is not sent until a search session begins when a user starts typing in the search widget.
Begin of search session (user enters text) =>action: "searchResultPage", source: "autocomplete" is recorded everytime a user begins to enter text into the search widget.
End of search session (user selects a search result) => action: "click"
End of search session (user closes search widget) => There are multiple ways this can be determined using SearchSatisfaction. Finding all search sessions that do not include a click action will show users that ended a session before selecting a search result. Also, if no search is performed within ten minutes of a previous search a new searchSessionid is generated. There is also a mwSessionId which should end when the browser has fully closed.

@Niedzielski, @MNeisler re: selecting the search widget - I think we can define an initiated search session by entering text. I'd be okay with removing that requirement.

Thank you, @MNeisler!

Do we need to track when someone clicks on the search widget without typing? If so, we will need this added in addition to the skin field.

@ovasileva, what do you think?

the requirements above

@ovasileva, can you confirm the requirements meet your needs? As I understand it, we're leaning on past work in SearchSatisfaction.

Do we also want to identify when the user ends their search session by submitting the form (i.e. not clicking on an autocomplete result)? AIUI this would be represented by a search session with:

  • An action: 'click' event with
    • A position property of -1 in the case where there were no autocomplete results or >= 0 otherwise
  • … immediately followed by an action: 'searchResultPage' event

@ovasileva, I think(?) the Discovery folks still need to be notified. Otherwise, if the requirements identified seem correct to you, I think this task can move to sign off.

@phuedx, @Niedzielski - would it make sense to close this task?

Per @Niedzielski above, I think Discovery need to be notified. I think that we also need to talk to them about extracting the sampling/bucketing logic from the instrument so that both teams can enable it as they see fit. Neither of these things block resolving this task though.

AFAICT the following question is unanswered:

Do we also want to identify when the user ends their search session by submitting the form (i.e. not clicking on an autocomplete result)? AIUI this would be represented by a search session with:

  • An action: 'click' event with
    • A position property of -1 in the case where there were no autocomplete results or >= 0 otherwise
  • … immediately followed by an action: 'searchResultPage' event

@phuedx, @Niedzielski - would it make sense to close this task?

Per @Niedzielski above, I think Discovery need to be notified. I think that we also need to talk to them about extracting the sampling/bucketing logic from the instrument so that both teams can enable it as they see fit. Neither of these things block resolving this task though.

AFAICT the following question is unanswered:

Do we also want to identify when the user ends their search session by submitting the form (i.e. not clicking on an autocomplete result)? AIUI this would be represented by a search session with:

  • An action: 'click' event with
    • A position property of -1 in the case where there were no autocomplete results or >= 0 otherwise
  • … immediately followed by an action: 'searchResultPage' event

This would be useful as we're currently interested in the behavior when it comes to the search button T259766: Data request for search button - it'd be nice to have this afterwards as well as a means of comparison. I would consider it in the nice to have category though, although it might be a higher priority for @alexhollender

Resolving this. Follow-up scheduled with the Search team next week.