[SPIKE] Define and document article criteria for article summarization content
Closed, ResolvedPublic5 Estimated Story PointsSpike
Actions

Assigned To

Authored By

	ovasileva
	Sep 23 2024, 8:33 AM

Project Tags

Referenced Files

	F57605179: top_10k_deduped
	Oct 10 2024, 10:06 PM

Subscribers

Description

Background

We would like to create a criteria, and through it a rough number of articles, for which we would like to generate summaries

User story

As a product team, we would like to define what a "good" set of articles for this experiment looks like, so that we can learn whether this feature would be helpful to readers

Requirements

Part 1: finalize criteria

Create a proposals for article summarization criteria (@ovasileva and @JScherer-WMF)
Review proposals with the engineers for feasibility
Select best criteria candidate
Based on the criteria, generate the list of articles to be used for the experiment

Part 2:
What is the process of ranking articles by popularity and subsetting them?
Document process
Part 3:
Generate list
This task was created by Version 1.2.0 of the Web team task template using phabulous

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Jdrewniak	T379708 [Hypothesis] [3.1.7] Article Summaries
Resolved		NBaca-WMF	T374143 [EPIC] Test Summarization and Simplification of Article Content
Resolved	Spike	ovasileva	T375363 [SPIKE] Define and document article criteria for article summarization content
Resolved	Spike	KSarabia-WMF	T375364 Generate article summaries for summary experiment

Event Timeline

ovasileva created this task.Sep 23 2024, 8:33 AM

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptSep 23 2024, 8:33 AM

ovasileva triaged this task as High priority.Sep 23 2024, 8:33 AM

ovasileva edited projects, added: Web-Team-Backlog-Archived (FY2024-25 Q2 Sprint 1); removed: Web-Team-Backlog-Archived.

ovasileva mentioned this in T375364: Generate article summaries for summary experiment.Sep 23 2024, 8:38 AM

ovasileva renamed this task from [SPIKE] Define article criteria for article summarization content to [SPIKE] Define and document article criteria for article summarization content.Sep 26 2024, 8:05 AM

Current proposal based on conversations with @JScherer-WMF and @Jdrewniak:
We are proposing to build an article pool from:

most popular articles screened for "good quality" using an API that Martin put together.
vital articles
featured articles

That should give us an article pool in the 10s of thousands. Then we rank those by popularity and take the top 10K.

ovasileva updated the task description. (Show Details)Sep 26 2024, 9:17 AM

ovasileva moved this task from Needs analysis to Ready for estimation on the Web-Team-Backlog-Archived (FY2024-25 Q2 Sprint 1) board.

ovasileva removed ovasileva as the assignee of this task.Sep 26 2024, 3:50 PM

ovasileva updated the task description. (Show Details)Sep 26 2024, 5:13 PM

note: might want to add criteria around article complexity and remove articles where we think summaries might be similar to the paragraph

Potentially split by/add as steps to this ticket:

Criteria
What is the process of ranking articles by popularity and subsetting them?
Generate list

ovasileva updated the task description. (Show Details)Sep 26 2024, 5:25 PM

ovasileva set the point value for this task to 5.

Jdlrobson moved this task from Ready for estimation to Ready for Development on the Web-Team-Backlog-Archived (FY2024-25 Q2 Sprint 1) board.Sep 26 2024, 5:27 PM

SToyofuku-WMF claimed this task.Oct 1 2024, 4:06 PM

SToyofuku-WMF moved this task from Ready for Development to Doing on the Web-Team-Backlog-Archived (FY2024-25 Q2 Sprint 1) board.

Just noting that we have multiple APIs that can help us retrieve lists based on categories like "featured" or "critical"
The categorymembers API looks something like this:

https://en.wikipedia.org/w/api.php?action=query&format=json&list=categorymembers&formatversion=2&cmtitle=%20Category%3AFeatured%20articles&cmnamespace=0&cmtype=page&cmlimit=500

and the API Sandbox is pretty useful for playing around with different API parameters.

For sorting the articles by popularity, we have the https://pageviews.wmcloud.org/ project which, when given a title, can return that articles pageviews on a daily, monthly or yearly timescale.
Looks like the API has been folded into the rest endpoint. e.g: monthly pageviews for the last 12 months for the article "Cat"
https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Cat/monthly/2023080100/2024083100

In regards to most popular articles, we keep those stats on https://stats.wikimedia.org
There is an API available that can provide the most popular articles on a given wiki for a given month:
https://wikimedia.org/api/rest_v1/metrics/pageviews/top/en.wikipedia.org/all-access/2024/09/all-days

When it comes to scoring popular articles by readability, we have an experimental API endpoint with Lift Wing:
https://api.wikimedia.org/wiki/Lift_Wing_API/Reference/Get_readability_prediction
This endpoint invokes an LLM so the response can be quite slow and resource intensive.

ovasileva moved this task from Backlog to Recommendations - summaries on the FY2024-25 KR 3.1 Content Discovery board.Oct 10 2024, 9:07 PM