Page MenuHomePhabricator

Measure whether users used the "did you mean" results that we serve to users somehow
Closed, ResolvedPublic

Description

We have no idea whether our users are actually using the "Did you mean" results we're serving them. Without quantitative data on this, it's hard to know whether any changes we make will actually serve our users better.

Let's figure out some way of measuring whether users are clicking on the did you mean results we serve them, so that we can perform improvements quickly and measure our success easily.

What I'm looking for to be recorded:

  • How many results the user was shown for their query
  • Whether the user was shown a did you mean result
  • Whether the user clicked on the did you mean result

Statistics that should be computable from this:

  • Percentage of queries where the user was shown a "Did you mean" result
  • Percentage of queries where, having been shown a "Did you mean" result, the user clicked on that result
  • Stakeholder: Search Team, Discovery Department
  • Benefit: We'll know how well any changes we make to "Did you mean" results actually work, as well as knowing current effectiveness
  • Estimate: To be added

Event Timeline

Deskana created this task.Jul 8 2015, 5:50 PM
Deskana updated the task description. (Show Details)
Deskana raised the priority of this task from to Normal.
Deskana added a subscriber: Deskana.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 8 2015, 5:50 PM
Deskana renamed this task from Measure effectiveness of the "did you mean" results that we serve to users somehow to Measure whether users used the "did you mean" results that we serve to users somehow.Jul 8 2015, 6:08 PM
Deskana updated the task description. (Show Details)
Deskana set Security to None.
Deskana updated the task description. (Show Details)
Deskana updated the task description. (Show Details)

Is EL the way to go for this collection?

If a user follows the "did you mean" link to a result, do we consider that a zero-result? Or do we only consider a dead-end to be a zero-result?

Is EL the way to go for this collection?

I think so. But, ultimately, as the product owner I am agnostic to the technical solution as long as it meets the acceptance criteria.

If a user follows the "did you mean" link to a result, do we consider that a zero-result?  Or do we only consider a dead-end to be a zero-result?

I would consider the user as having done two searches. The first gave them zero results and the "did you mean". The second one was based on the "did you mean", and may or may not have given them results. Does that answer your question?

Ah, that makes sense, thanks.

Change 232376 had a related patch set uploaded (by EBernhardson):
Measure usage of Special:Search did you mean

https://gerrit.wikimedia.org/r/232376

Change 232376 merged by jenkins-bot:
Measure usage of Special:Search did you mean

https://gerrit.wikimedia.org/r/232376

Deskana closed this task as Resolved.Sep 9 2015, 2:36 AM
Deskana moved this task from Done to Resolved on the Discovery-Search (Current work) board.