Page MenuHomePhabricator
Create Task
Maniphest T374965

Analyze empty state of search experiment
Closed, ResolvedPublicDesign
Actions

Description

Background

See instrumentation details in T372780: Instrument empty state of search experiment

User story

  • As a product team, we want to know whether adding recommendations into search will benefit users so that we know whether to invest our time into this idea

Requirements

  • Report on the overall clickthrough rate for search recommendations coming from the browser extension

CTR data is on dashboard: https://grafana-rw.wikimedia.org/d/d3d538c9-00c3-4b4f-9e79-71cfcb3f0c75
Report enwiki, eswiki, and enwiki+eswiki for a 2-week timespan

  • Compare to clickthrough rate for related pages on desktop, mobile

Related Objects
Search...

Event Timeline

ovasileva created this task.Sep 17 2024, 2:25 PM
ovasileva triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2024, 2:25 PM
ovasileva moved this task from Incoming to Analyst Consultation on the Web-Team-Backlog-Archived board.Sep 17 2024, 2:25 PM
ovasileva edited projects, added Web-Team-Backlog-Archived (FY2024-25 Q1 Sprint 6); removed Web-Team-Backlog-Archived.
ovasileva assigned this task to jwang.Sep 17 2024, 3:20 PM
jwang added a project: Product-Analytics (Kanban).Sep 18 2024, 11:50 PM
jwang updated the task description. (Show Details)Sep 23 2024, 5:43 PM
jwang updated the task description. (Show Details)Sep 23 2024, 5:47 PM
jwang updated the task description. (Show Details)Sep 23 2024, 6:02 PM
jwang updated the task description. (Show Details)
Jdlrobson moved this task from Needs analysis to Doing on the Web-Team-Backlog-Archived (FY2024-25 Q1 Sprint 6) board.Sep 25 2024, 5:59 PM
jwang added a subscriber: Jdrewniak.Sep 26 2024, 12:06 AM

Hi, @Jdrewniak, I'd like to confirm the data in the dashboard to calculate CTR rate.

  • For impression of suggested articles on enwiki, can I use the data from Total daily impressions across all wikis in below chart?
  • For clicks on suggested articles on enwiki , can I use the data from Total daily clicks across all wikis in below chart?
  • What is Total daily clickAdditional across all wikis for? Should it also be counted as part of the clicks?

image.png (598×1 px, 95 KB)

Jdrewniak added a comment.Sep 26 2024, 4:59 PM
This comment was removed by Jdrewniak.
Jdrewniak added a comment.Sep 26 2024, 5:09 PM

@jwang yes, my apologies, I mislabled the legend in that chart title "Empty search experiment (enwiki)" but it does point to enwiki specific data.

The statsd data streams are divided on a per-wiki basis, so for stats specifically for enwiki, we can use the following datasets. I used Graphite to query this data for the charts.

  • MediaWiki.experiment.web.emptySearch.enwiki.impression.count - the number of search suggestion impressions
  • MediaWiki.experiment.web.emptySearch.enwiki.enabled.count - the number of pageviews with the extension enabled
  • MediaWiki.experiment.web.emptySearch.enwiki.click.count - the number of search suggestion clicks.
  • MediaWiki.experiment.web.emptySearch.enwiki.clickAdditional.count -This can be grouped in with clicks, these are basically right-clicks on search suggestions.
jwang added a comment.EditedSep 27 2024, 6:07 PM

Analysis Summary

  • Report on the overall clickthrough rate for search recommendations coming from the browser extension
    • On English wikipedia (enwiki) desktop, the overall CTR for search recommendation is 1.53%
    • On Spanish wikipedia (eswiki) desktop, the overall CTR for search recommendation is 0.72%
    • The overall CTR across both wikis is 1.44%.

Daily trend of English Wikipedia (enwiki)

image.png (508×1 px, 74 KB)

Daily trend of Spanish Wikipedia (eswiki)

image.png (572×1 px, 79 KB)

  • Compare to clickthrough rate for related pages on desktop, mobile
    • We observed that the CTR for search recommendations is higher than that for related pages on desktop for both English Wikipedia and Spanish Wikipedia
      • English Wikipedia (enwiki): 1.53% vs 0.41% , diff 1.12 pp
      • Spanish Wikipedia (eswiki): 0.72% vs 0.13%, diff 0.59pp
Average daily CTR for related pagesdesktopmobile web
English Wikipedia (enwiki)0.41%4.91%
Spanish Wikipedia (eswiki)0.13%5.74%

Note: The above summary is based on the data collected between 2024-09-10 and 2024-09-24. Please find the detailed data in the Google Sheet.

Jdlrobson edited projects, added Web-Team-Backlog-Archived; removed Web-Team-Backlog-Archived (FY2024-25 Q1 Sprint 6).Oct 1 2024, 10:10 PM
Jdlrobson moved this task from FY2024-25 Q1 Sprint 6 to Analyst Consultation on the Web-Team-Backlog-Archived board.
jwang moved this task from Next 2 weeks to Needs Review on the Product-Analytics (Kanban) board.Oct 3 2024, 5:28 PM
ovasileva moved this task from Backlog to Recommendations - search on the FY2024-25 KR 3.1 Content Discovery board.Oct 10 2024, 9:07 PM
dr0ptp4kt subscribed.Oct 15 2024, 6:40 PM
jwang mentioned this in T376817: Analyze results of article recommendation experiment.Oct 22 2024, 8:53 PM
mpopov mentioned this in T355837: Add Prometheus support to statsd.js via mw.track().Oct 24 2024, 6:00 PM
Jdlrobson edited projects, added Web-Team; removed Web-Team-Backlog-Archived.Nov 13 2024, 11:24 PM
Jdlrobson moved this task from Incoming to Freezer on the Web-Team board.Nov 13 2024, 11:28 PM
NBaca-WMF moved this task from Freezer to Incoming on the Web-Team board.Nov 18 2024, 4:01 PM
ovasileva closed this task as Resolved.Nov 18 2024, 4:06 PM

All done! Resolving

Ottomata mentioned this in T390328: Enable querying operational (prometheus) metrics via the WMF Data Platform.Mar 28 2025, 8:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits