Page MenuHomePhabricator

Build dashboards for search activity on MediaSearch on Commons
Open, MediumPublic

Description

In order to better understand how the new MediaSearch on Commons behaves, we should have one (or multiple) dashboard(s) that allow us to monitor search metrics. The metrics we're interested in are:

  • Number of searches made
    • This will help us learn how often search being used so that we can see if usage of the feature is growing.
  • Number of search sessions
    • This will help us learn how often search being used so that we can see if usage of the feature is growing
  • Number of searches per session
    • This will help indicate how long it takes the user to find what they're looking for.
  • Search session length
    • This will help indicate how long it takes the user to find what they're looking for.
  • Click-through rate (to quickview and from quickview to file pages)
    • This will let us know how often they're clicking through to results to give a sense of whether they're finding what they're looking for
  • Average position of clicked result in successful searches
    • This will help us understand the impact of the grid on search results, and we may make changes to the grid based on this information.
  • How often concept chips are clicked/what percentage of users click a concept chip during their search session
    • This will let us know how often concept chips are being used, so that we can determine the success of the feature.
  • What percentage of users use a filter during their search session.
    • This will let us know how often filters are being used, so that we know the success of the feature.
  • What percentage of users use more than 1? 2? 3? filters in their session
  • What filters are used the most
    • This will help us understand which filters are most useful so that we can plan features accordingly.
  • What percentage of users/how many users use the functionality to copy the filename or wikitext to their clipboard
    • How people are using the functionality to copy the filename or wikitext can give us a signal as to how often they're reusing the images in other wikimedia projects.

We're interested in having this for both the existing search on Commons as well as MediaSearch, and might create specific sub-tasks for each.

Event Timeline

nettrom_WMF added subscribers: Ramsey-WMF, CBogen.

@Ramsey-WMF @CBogen @Abit Can you clarify Structured Data's priority & timing for this request?

  • Are the metrics needed prior to a launch? If so, what is your planned launch date?
  • Are their other planned actions or decisions that depend on the metrics? Will not having the metrics block work or launches?

@kzimmerman

  • Baseline metrics for the old search are what is time sensitive. We are hoping to have those by early August. That is approximately when we will be ready to launch an A/B test comparing the old results to the new results.
  • Our OKRs depend heavily on the metrics for MediaSearch, and will help us make decisions going forward about the launch. We have a bit more time for that, as we're planning to launch in Q2.
LGoto triaged this task as Medium priority.Jul 20 2020, 4:08 PM
LGoto moved this task from Triage to Needs Investigation on the Product-Analytics board.
CBogen added a comment.Aug 3 2020, 5:33 PM

Hi @nettrom_WMF - position is listed in the schema, but not listed in this ticket - can you confirm that we're planning to measure position for MediaSearch and that we'll have a baseline for position in the current Commons search? Thanks!

nettrom_WMF updated the task description. (Show Details)Aug 4 2020, 12:15 AM
nettrom_WMF updated the task description. (Show Details)Aug 4 2020, 12:19 AM

Hi @nettrom_WMF - position is listed in the schema, but not listed in this ticket - can you confirm that we're planning to measure position for MediaSearch and that we'll have a baseline for position in the current Commons search? Thanks!

Yes, I want to have position in the schema for Media Search as I expect that to be something we'll be investigating down the line. I've added "average position of clicked result in successful searches" to the list of metrics above, as I suspect that'll be one of the metrics you're interested in for comparing backends in legacy search. While I'm not certain how we'd go about making comparisons of positions in a grid layout as the research wasn't clear on that (there are lots of complicating factors), I'm thinking that we can consult with Miriam on that when the time comes.

CBogen renamed this task from Build dashboards for search activity on Commons to Build dashboards for search activity on MediaSearch on Commons.Oct 6 2020, 4:15 PM
CBogen updated the task description. (Show Details)
CBogen updated the task description. (Show Details)
CBogen updated the task description. (Show Details)Oct 20 2020, 8:46 PM

I've made T258183 a subtask of this task, since we can't build dashboards until the instrumentation is in place.