Page MenuHomePhabricator

Identify metrics for search UI recommendations experiment
Closed, ResolvedPublicSpike

Description

Background

  • We will be testing the UI for adding recommendations into search. This ticket will cover the metrics we want to measure as part of this experiment

User story

  • As a product team, we want to know whether adding recommendations into search will benefit users so that we know whether to invest our time into this idea

Requirements

  • Brainstorm and finalize list of metrics for UI recommendations experiment
  • Make a list of required metrics and nice to have metrics
  • Document outcomes

Event Timeline

ovasileva created this task.
Restricted Application changed the subtype of this task from "Design" to "Spike". · View Herald TranscriptJul 29 2024, 10:32 AM

Baseline metrics set 1:

Quantitative metrics - high priority

  • Search sessions initiated
  • Impressions of search suggestion
  • Clicks on suggestions
  • Text types into search - i.e. we would want some way to know whether a session was completed using a suggestion, completed using search (they began typing and selected a result), or abandoned

Qualitative metrics - nice to have

  • Overall interest in search suggestions
  • Overall satisfaction with UI of feature
  • Ease of use of functionality itself

Baseline metrics redacted:

Quantitative metrics - high priority

Search sessions started
Clicks on suggestions from extension
Type of session - i.e. we would want some way to know whether a session was completed using a suggestion, completed using search (they began typing and selected a result), or abandoned

Qualitative metrics - nice to have

Overall interest in search suggestions
Overall satisfaction with UI of feature
Ease of use of functionality itself

@jwang to put together a spec based on the above

After discussed with @ovasileva, our goal is to gain a sense of user preferences with minimal instrumentation. The major question we want to understand is "Are people interacting with the prototype suggestion and how frequently? " Based on these, I have drafted an instrumentation spec: link.

@KSarabia-WMF , let me know if you have any questions.

ovasileva renamed this task from Identify baseline metrics for search UI recommendations experiment to Identify metrics for search UI recommendations experiment.Sep 2 2024, 10:40 AM

@KSarabia-WMF - assigned this to you for QA. The expectation here is to sign off if no open questions on the spec that are not covered in T372780: Instrument empty state of search experiment remain.

Jdlrobson lowered the priority of this task from High to Medium.
Edtadros added subscribers: Jdlrobson, Edtadros.

@Jdlrobson, I'm not sure what to test with this, my initial thought is that it's a spike so there isn't any QA on my part. But in the event my initial thought is wrong please provide some test steps.