Page MenuHomePhabricator

Special:Search Quickview analytics
Closed, ResolvedPublic

Description

As a product manager , I want to understand how the new QuickView features are being used and how they impact search metrics in general so that I can make decisions about how to build and/or make changes to the features going forward and determine their success.

This ticket is to add metrics about QuickView to the Search Improvements dashboard in the parent task.

Metrics to measure:

  • Usage Metrics:
    • Number (%) of users opt in/out quickview
    • Number of quickview open per session
    • Number (%) of sessions (searches) using quickview
  • Interaction Metrics:
    • CTR on titles/sections/images inside quickview panel
    • Click through position of the search result which open previews
  • Other related metrics:
    • CTR on special:search results
    • Number of searches (per session)

Dashboard: Search Previews

Event Timeline

mpopov triaged this task as Medium priority.

Is there anything new that we'll need to instrument? If so, any guidance would be appreciated - what kind of data would need capturing?

Is there anything new that we'll need to instrument? If so, any guidance would be appreciated - what kind of data would need capturing?

My understanding is that's covered in T307572, but @cchen can you update that ticket with the status so that we can move it forward?

Please find the stats for the search preview on desktop in January below. The data is from 01/05/2023 across three Wikipedia: Indonesian, Portuguese, and Russian.

Here's a summary of the data:

  • Very few users opt out the search preview on all 3 wikis.
  • Indonesian Wikipedia has the highest open rate of search previews among all 3 wikis. The open rate on Russian Wikipedia is the lowest.
  • The snippet section in the search preview has the highest CTR on all 3 wikis, followed by the image section and article sections. This may be because of the sequences of each section in the preview. In addition, not every preview of the search result has images, or article sections (but most of them have snippets).
  • The average position of search results with previews opened is 0 (which is the first result) on all 3 wikis.
  • To compare with the virtual pageview on Special:Seach page, use the charts mentioned in here: T324852#8624174. In January, users are still use virtual pageviews more often on all three wikis. The amount of pageview previews was ~48% less than virtual pageviews on Indonesian Wiki, ~58% less on Portuguese Wiki, and ~80% less on Russian Wikipedia.

Opt-out rate
Calculated by dividing the total number of users with opt-out search previews by the total number of all registered users on the wiki.

  • Indonesian Wikipedia: 0.01%
  • Portuguese Wikipedia: 0%
  • Russian Wikipedia: 0.01%

Number of previews per day:

  • Indonesian Wikipedia: 347.4
  • Portuguese Wikipedia: 763.9
  • Russian Wikipedia: 1144.3

% of search sessions with previews opened
Defined as the percentage of search sessions where the user clicked at least one search preview.

  • Indonesian Wikipedia: 8.0%
  • Portuguese Wikipedia: 6.3%
  • Russian Wikipedia: 3.7%

Number of previews opened per session
This uses search sessions where the user opened at least one search preview.

  • Indonesian Wikipedia: 1.6
  • Portuguese Wikipedia: 1.5
  • Russian Wikipedia: 1.4

Click-through rates

Defined as the proportion of search sessions with clicks on sections in search previews divided by search sessions where the user opened at least one search preview.

SnippetArticle SectionsImageInterwiki Commons
Indonesian Wikipedia16.6%2.3%2.9%1.8%
Portuguese Wikipedia19.3%3.5%3.7%1.9%
Russian Wikipedia15.4%2.1%1.3%1.1%

Average position of clicked result
Defined as the median position of a clicked result where the user opened at least one search preview. We use the median, as the long-tailed distribution renders the mean.

  • Indonesian Wikipedia: 0
  • Portuguese Wikipedia: 0
  • Russian Wikipedia: 0

Below is a summary of February data for search previews, the detailed number can be found in this dashboard

  • Very few users opt-out the search preview across 3 wikis, the opt-out rates are all below 0.01%.
  • The number of previews opened on mobile and the open rate is twice as much as on desktop for Indonesian Wikipedia and Portuguese Wikipedia. And is above 15% on mobile in Indonesian Wikipedia in February and March. For Russian Wikipedia, these numbers are almost triple on mobile. This may be because the virtual pageviews are not available on mobile. and search preview is helping users to quickly discover information on mobile.
  • Indonesian Wikipedia has the highest open rate (around 8% on desktop, 15% on mobile) of search previews among all 3 wikis on both desktop and mobile, Russian Wikipedia is the lowest (3.8% on desktop and 9% on mobile).
  • Portuguese Wikipedia started to have an increasing daily number of previews opened and an increasing open rate on desktops.
  • The number of previews opened per session on mobile is 30% higher on mobile than on desktop in all 3 wikis.
  • The click-through rates of all the sections in the search preview are higher on mobile than on desktop in all 3 wikis. Users are more engaged with search previews on mobile platforms. Especially on Russian Wikipedia, the CTR for the image section (1.2% on desktop, 3.9% on mobile) and article section (2.4% on desktop and 4% on mobile) are much higher compared with the other 2 wikis.
  • Portuguese Wikipedia has the highest click-through rate on the section on both desktop and mobile except the mobile interwiki commons.
  • The median click positions are 0 (first result) on mobile and desktop on Portuguese Wikipedia. On Russian Wikipedia and Indonesian Wikipedia, the median click positions are 0 (the first result) on desktop and 1 (the second result) on mobile.
cchen updated the task description. (Show Details)

update a summary of June to July search previews data with the additional wikis added, the detailed number can be found in this dashboard

  • Opt-out Rate: Very few users opt-out the search preview across wikis, the opt-out rates are all below 0.01%.
  • Number of Previews Opened: The numbers of previews opened on mobile are twice as much as on desktop for idwiki, ptwiki, huwiki. For ruwiki and ukwiki, these numbers are almost triple on mobile. On nlwiki, the number of previews is 15% less on mobile than on desktop. And on cawiki and nowiki, these numbers are similar on desktop and mobile.
  • % of Sessions with Preview Opened: The percentage of search sessions where the user clicked at least one search preview are almost doubled on mobile compared to on desktop across all the wikis. Idwiki has the highest open rate on desktop (~7.4%) and ptwiki has the highest open rate on mobile (~14.1%). Ruwiki has the lowest open rate among all the wikis on both desktop (~3.6%) and mobile (~8.1%).
  • Average number of Search Previews opened: The number of previews opened per session on mobile is 30% higher on mobile than on desktop in all 3 wikis.
  • Click-through rates: the click through rates of all the sections in the search preview are still higher on mobile than on desktop across all 8 wikis.
    • Snippet section has the hightest click-through rate, while all other elements have click-through rate below 5%.
    • Interwiki Commons started to have a large increase in click-through rate from March on ptwiki and idwiki compare to previous months.
    • ptwiki has the highest click-through rates on the section on both desktop and mobile except the mobile interwiki commons and interwiki links.
    • Ukwiki and huwiki have the lowest click-through rates on most of the sections among all the wikis.
  • Average Click Position: The average click positions are 0 (first result) on desktop and 1 (the second result) on mobile.