Search metrics on Wikimedia Commons (Redux)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• chelsyx
	Feb 20 2018, 8:09 PM

Description

Background

In T177534, in order to understand how well our search engine serve the users on Commons, we computed several search metrics with event logging data (TestSearchSatisfaction2 table) in November 2017, and compare them with English Wikipedia. Specifically, we computed:

Zero results rate: Proportion of searches that did not yield any results. The lower the better.
Clickthrough rate: Proportion of searches with at least one click on the search results. The higher the better.
Proportion of searches with clicks to see other pages of the search results. The lower the better.

We didn't investigate search on mobile, since we have very few eventlogging data on mobile web on Commons (less than 100 search result pages daily). The codebase for this analysis is on GitHub.

Objective

Your task is to re-do the analysis and answer the following questions:

Compare the desktop full-text search zero result rate on Commons vs English Wikipedia. Break down the number by day.
Compare the desktop full-text search clickthrough rate on Commons vs English Wikipedia. Break down the number by day.
Compare the proportion of desktop full-text searches with clicks to see other pages of the search results on Commons vs English Wikipedia. Break down the number by day.
Compare the dwell time on articles after users clickthrough on Commons vs English Wikipedia.
Compare the desktop autocomplete search clickthrough rate on Commons vs English Wikipedia. Break down the number by day.

Feel free to use different data or presentation to answer the question, or figure out additional insights that were not in the original report.

Related Objects
Search...

Status	Assigned	Task
Resolved	MNeisler	T185363 [EPIC] Learn about our databases and tools
Resolved	MNeisler	T187827 Search metrics on Wikimedia Commons (Redux)
Resolved	MNeisler	T188421 Investigate the full-text desktop search CTR decline on Wikimedia Commons

Event Timeline

• chelsyx created this task.Feb 20 2018, 8:09 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 20 2018, 8:09 PM

• chelsyx triaged this task as Medium priority.Feb 20 2018, 8:10 PM

• chelsyx added a parent task: T185363: [EPIC] Learn about our databases and tools.

• chelsyx mentioned this in T185363: [EPIC] Learn about our databases and tools.

MNeisler mentioned this in T186575: File type and deletion metrics on Wikimedia Commons (Redux).Feb 21 2018, 9:40 PM

Reproduced. See initial codebase and output: https://github.com/MeganNeisler/SDoC-Baseline-Metrics-Redux/tree/master/T187827. Any feedback and suggestions are welcome!

Good job @MNeisler !

Some comments:

The decline of full-text search clickthrough rate on Commons is an interesting finding! I created a ticket T188421 for you to further investigate this issue.
For those daily breakdown graphs, your subtitles are "Dashed line marks the average ...". The word "average" is misleading here. Instead of the average of daily ZRR/CTR, the dashed line represents an "overall" rate, which is computed with all the data points in your whole dataset.
In the section Dwell time on articles after users clickthrough:
- It would be better if you use the word "page" instead of "article", since the search results could be in many other namespaces.
- "A higher proportion of Commons articles have a user dwell time rate of under 10 seconds." Are you trying to say "Visitors are more likely to left the page within 10 seconds on Commons"?
- The facet labels on the second graph should be change to: "0s" -> "0-10s", "10s" -> "10-20s", etc. Also it would be better to put the legends on the bottom, so the x axis labels would be well separated.
In the section Desktop autocomplete search clickthrough rate:
- "Commons has lower rate of clickthrough rates for autocompleted searches compared on English Wikipedia on desktop" -> "Commons has lower rate of clickthrough for autocompleted searches compared to English Wikipedia on desktop"
- Remove the data of the last day from the second chart, because we don't have all the data from that day. Aggregating incomplete daily data would result in distorted metric and very large CI.