Page MenuHomePhabricator

Search metrics on Wikimedia Commons (Redux)
Closed, ResolvedPublic

Description

Background

In T177534, in order to understand how well our search engine serve the users on Commons, we computed several search metrics with event logging data (TestSearchSatisfaction2 table) in November 2017, and compare them with English Wikipedia. Specifically, we computed:

  • Zero results rate: Proportion of searches that did not yield any results. The lower the better.
  • Clickthrough rate: Proportion of searches with at least one click on the search results. The higher the better.
  • Proportion of searches with clicks to see other pages of the search results. The lower the better.

We didn't investigate search on mobile, since we have very few eventlogging data on mobile web on Commons (less than 100 search result pages daily). The codebase for this analysis is on GitHub.

Objective

Your task is to re-do the analysis and answer the following questions:

  • Compare the desktop full-text search zero result rate on Commons vs English Wikipedia. Break down the number by day.
  • Compare the desktop full-text search clickthrough rate on Commons vs English Wikipedia. Break down the number by day.
  • Compare the proportion of desktop full-text searches with clicks to see other pages of the search results on Commons vs English Wikipedia. Break down the number by day.
  • Compare the dwell time on articles after users clickthrough on Commons vs English Wikipedia.
  • Compare the desktop autocomplete search clickthrough rate on Commons vs English Wikipedia. Break down the number by day.

Feel free to use different data or presentation to answer the question, or figure out additional insights that were not in the original report.

Event Timeline

Good job @MNeisler !

Some comments:

  • The decline of full-text search clickthrough rate on Commons is an interesting finding! I created a ticket T188421 for you to further investigate this issue.
  • For those daily breakdown graphs, your subtitles are "Dashed line marks the average ...". The word "average" is misleading here. Instead of the average of daily ZRR/CTR, the dashed line represents an "overall" rate, which is computed with all the data points in your whole dataset.
  • In the section Dwell time on articles after users clickthrough:
    • It would be better if you use the word "page" instead of "article", since the search results could be in many other namespaces.
    • "A higher proportion of Commons articles have a user dwell time rate of under 10 seconds." Are you trying to say "Visitors are more likely to left the page within 10 seconds on Commons"?
    • The facet labels on the second graph should be change to: "0s" -> "0-10s", "10s" -> "10-20s", etc. Also it would be better to put the legends on the bottom, so the x axis labels would be well separated.
  • In the section Desktop autocomplete search clickthrough rate:
    • "Commons has lower rate of clickthrough rates for autocompleted searches compared on English Wikipedia on desktop" -> "Commons has lower rate of clickthrough for autocompleted searches compared to English Wikipedia on desktop"
    • Remove the data of the last day from the second chart, because we don't have all the data from that day. Aggregating incomplete daily data would result in distorted metric and very large CI.
mpopov added a subscriber: mpopov.

Good job, @MNeisler!