[Analysis] Analyze summarization experiment
Closed, ResolvedPublic0 Estimated Story Points
Actions

Assigned To

Authored By

	ovasileva
	Dec 2 2024, 10:50 AM

Description

Background

We would like to perform the analysis on the summarization experiment to see how users are interacting with the summaries

User story

As a product team, we want to measure how users are interacting with the summaries, so that we can see if the feature meets its success criteria and should be worked on further

Requirements

Answer the following questions:

How frequently are summaries opened? - Out of all pageviews where summaries are displayed, X% of pageviews had a summary opened?
Do users find the summaries useful?
- What percentage selected "yes"
- What percentage selected "no"
- What was the "yes" to "no" ratio?

BDD

For QA engineer to fill out

Test Steps

For QA engineer to fill out

Design

Acceptance criteria

Complete all requirements

This task was created by Version 1.2.0 of the Web team task template using phabulous

Related Objects
Search...

Status	Assigned	Task
Resolved	Jdrewniak	T379708 [Hypothesis] [3.1.7] Article Summaries
Resolved	NBaca-WMF	T374143 [EPIC] Test Summarization and Simplification of Article Content
Resolved	Jdlrobson	T374633 [EPIC] Run Figma Prototype experiment soliciting user feedback on lead paragraph summarization
Resolved	JScherer-WMF	T374636 Create an interactive Figma prototype of summarization for user testing
Resolved	JScherer-WMF	T374638 Setup and run Userlytics test of article summarization
Resolved	ovasileva	T374635 Precompute Article Summaries
Resolved	jwang	T374639 Identify and document success metrics for Summarization, create spec for instrumentation
Resolved	ovasileva	T370212 [Experiment] Design the reading experience of simplified article summaries
Resolved	ovasileva	T378098 Deploy summary experiment
Open	ovasileva	T376903 [Spike] How to Instrument metrics for summarization experiment
Resolved	ovasileva	T381253 [Analysis] Analyze summarization experiment

Event Timeline

ovasileva created this task.Dec 2 2024, 10:50 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 2 2024, 10:50 AM

Marking as stalled until the experiment has ran for at least two weeks to make sure we have enough data

jwang added a project: Product-Analytics.Dec 2 2024, 7:01 PM

ovasileva merged a task: T380647: [analytics] Simple summary experiment - Analyze results.Dec 5 2024, 9:46 AM

ovasileva added a subscriber: Jdrewniak.

jwang edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Dec 13 2024, 9:24 PM

jwang moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Dec 19 2024, 6:18 PM

Analysis Summary

How frequently are summaries opened?

Out of all pageviews where summaries are displayed, 8.09% of pageviews had a summary opened.
The pages containing the summary received 10,198 impressions and 825 total clicks to open the summary.

Do users find the summaries useful?

A total of 101 responses were received in our survey.
75.2% of all clicks selected "yes"
24.8% of all clicks selected "no"
The "yes" to "no" ratio is 3.0

The daily trend of English Wikipedia (enwiki)

Note: The above summary is based on data collected from the English Wikipedia (enwiki) desktop platform between December 4, 2024, and December 17, 2024. Only users who had enabled the extension could view the summary, and not all pages included one. Detailed data is available in the Google Sheet.

@ovasileva, @Jdrewniak, let me know if you have any questions.

SToyofuku-WMF subscribed.Dec 20 2024, 5:30 PM

ovasileva changed the task status from Stalled to In Progress.Jan 6 2025, 2:07 PM

Thanks @jwang! Moving to spring 1 for documentation purposes and assigning to myself

ovasileva moved this task from Incoming to Sprint Backlog on the Web-Team board.Jan 10 2025, 7:43 AM

ovasileva moved this task from Sprint Backlog to FY2024-25 Q2 Sprint 6 (Dec 16 - Jan 13, 2025) on the Web-Team board.Jan 13 2025, 11:32 AM

ovasileva edited projects, added Web-Team (FY2024-25 Q2 Sprint 6 (Dec 16 - Jan 13, 2025)); removed Web-Team.

ovasileva moved this task from FY2024-25 Q2 Sprint 6 (Dec 16 - Jan 13, 2025) to Sprint Backlog on the Web-Team board.

ovasileva edited projects, added Web-Team; removed Web-Team (FY2024-25 Q2 Sprint 6 (Dec 16 - Jan 13, 2025)).

Jdlrobson moved this task from Sprint Backlog to Incoming on the Web-Team board.Jan 13 2025, 5:32 PM

Jdlrobson moved this task from Incoming to Q3 on the Web-Team board.Jan 13 2025, 6:02 PM

Jdlrobson moved this task from Q3 to Sprint Backlog on the Web-Team board.Jan 13 2025, 6:35 PM

Jdlrobson moved this task from Sprint Backlog to Q3 Sprint 1 (Jan 14 - Jan 24, 2025) on the Web-Team board.Jan 14 2025, 5:51 PM

Jdlrobson edited projects, added Web-Team (Q3 Sprint 1 (Jan 14 - Jan 24, 2025)); removed Web-Team.

Jdlrobson moved this task from Ready for Development to Sign-Off on the Web-Team (Q3 Sprint 1 (Jan 14 - Jan 24, 2025)) board.Jan 15 2025, 5:16 PM

Results now published in https://www.mediawiki.org/wiki/Reading/Web/Content_Discovery_Experiments/Summaries. Resolving this.

Jdrewniak mentioned this in T386491: [summaries] Experiment design & measurement plan.Feb 14 2025, 5:04 PM

	F58030730: image.png
	Dec 19 2024, 6:39 PM

[Analysis] Analyze summarization experimentClosed, ResolvedPublic0 Estimated Story PointsActions