Page MenuHomePhabricator

Follow-up from Metrics
Closed, ResolvedPublic

Description

There was a question that came up in regards to the Quarterly Metrics meeting that was held on Oct 31, 2017.

  • what is the clickthrough rate for mobile
  • we presented it for desktop only (see screen capture of what was presented)

Event Timeline

debt created this task.Oct 31 2017, 11:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 31 2017, 11:28 PM
debt assigned this task to chelsyx.Nov 1 2017, 8:30 PM
debt updated the task description. (Show Details)
debt removed a subscriber: chelsyx.

On the quarterly metrics meeting, we show the clickthrough rate on desktop by search type: fulltext ~21.3%, autocomplete ~95%. This is the proportion of pageviews that have at least one clickthrough. For fulltext search, each search result page is a pageview; for autocomplete, as long as the users don't clickthrough, no matter how many queries users put in the search box, the pageview ID remains the same.

However, we don't track the pageview ID on mobile currently (we are using the MobileWebSearch and MobileWikiAppSearch schema to log search events on mobile). So we cannot compute the proportion of pageviews that have at least one clickthrough on mobile. The following clickthrough rates are total number of clicks divided by the total number of search results page (for autocomplete search, every single extra character users type will initiate a new search request, and return a new search result page if we can find a match).

For search on mobile web, our eventlogging schema tracks autocomplete search only. The clickthrough rate is about 13%.

For search on mobile app, we can't separate the clickthrough rate by fulltext and autocomplete search, since the schema doesn't log the type of search for click event. The clickthrough rate is about 30.6%.

For comparison, I calculate the clickthrough rate on desktop using the same method too (total number of clicks / total number of search results page). For autocomplete search, it's about 24.6%; for full-text search, it's about 21.9%.

debt added a comment.Nov 7 2017, 3:10 PM

Follow-up Q&A (via email) from @chelsyx:

On desktop, ~95% of searches lead to a click on an autocomplete?

95% of pageviews lead to a click. Precisely speaking, this is not equal to 95% of searches. For example, in the search box on https://en.wikipedia.org/wiki/Main_Page, first I search for "cat" but didn't clickthrough, then I search for "dog" and clickthrough. In this case, 100% of pageviews lead to a click, but only 50% of searches lead to a click. However, in autocomplete, since search requests are sent while typing, it's difficult to define how many searches happened on a pageview. Therefore, we can say ~95% of autocomplete searches lead to a click, but the actual number should be smaller.

On mobile web, we don't know for sure but the best proxy we have (searches/clicks) seems to indicate that different things are happening on mobile and desktop as these proxies are very different (13% mobile web v. 23/24% desktop) ~30% on apps (Full text search is really hard to get to--maybe doesn't even exist?)

Jon, actually on app, 36% of search result pages are marked as full-text. But it seems app would automatically switched to fulltext mode if users got no results from autocomplete, so maybe the user experience between full-text and autocomplete on app is not as distinct as on mobile web/desktop? Unfortunately we don't log the type of search for click event on app so I don't know if there is any difference in clickthrough rate between this two mode. Maybe the apps team have an answer.
As to the difference between apps and mobile web, it could be the browsers, instrumentation, or other factors. For example, on mobile web autocomplete search, if we don't find a match for the query, a search result page will still be generated, while for apps it just returns nothing. This results in higher number of search result page and lower CTR for mobile web than apps. However, after removing the zero result search result pages for mobile web, it's CTR is 16%, which is still much lower than apps' 30%.

debt closed this task as Resolved.Nov 9 2017, 10:13 PM