Maniphest T320606

Create bulk ingestion articles endpoint
Closed, ResolvedPublic13 Estimated Story Points
Actions

Assigned To

Authored By

	Protsack.stephan
	Oct 12 2022, 9:29 AM

Description

We need to create articles bulk ingestion articles endpoint to ingest pages from all the the projects that we support so that we can then use that data for On-demand and Snapshots service.

Acceptance criteria
I can trigger articles endpoint for a project name and ingest all of the articles in that project.

To-Do

add articles method to bulk.proto in protos repository
add articles handler to bulk-ingestion service
- input params should be a project and namespace
- need to investigate if titles dump is still the best way to get a list of articles
- should produce an event with list of titles to go through

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		JArguello-WMF	T315299 Wikimedia Enterprise API(s) V2
Resolved		JArguello-WMF	T300839 Snapshots V2
Resolved		Anribolon	T320606 Create bulk ingestion articles endpoint
Resolved		Anribolon	T324184 Implement bulk ingestion handler in Structured Data service
Resolved		REsquito-WMF	T325144 Get templates in a page
Resolved		REsquito-WMF	T325145 Get categories in a page
Resolved	BUG REPORT	REsquito-WMF	T325274 Text Processor service GetDictWords endpoint can't sustain load
Resolved	BUG REPORT	prabhat	T325269 Text Processor service GetDictWords endpoint is slow
Resolved		Protsack.stephan	T325294 Add abstract field to the article

Event Timeline

Protsack.stephan created this task.Oct 12 2022, 9:29 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 12 2022, 9:29 AM

Protsack.stephan added a parent task: T300839: Snapshots V2.Oct 12 2022, 9:30 AM

Protsack.stephan added a project: Wikimedia Enterprise Engineering.Oct 12 2022, 9:36 AM

Protsack.stephan moved this task from To Be Estimated/To Be Discussed to Estimated /Discussed on the Wikimedia Enterprise board.Oct 20 2022, 9:33 AM

Protsack.stephan claimed this task.Oct 20 2022, 12:08 PM

Protsack.stephan triaged this task as High priority.

Protsack.stephan updated the task description. (Show Details)

Protsack.stephan set the point value for this task to 13.Oct 20 2022, 2:45 PM

Protsack.stephan reassigned this task from Protsack.stephan to Anribolon.Oct 21 2022, 11:21 AM

Anribolon moved this task from Estimated /Discussed to In Progress on the Wikimedia Enterprise board.Oct 24 2022, 1:42 PM

AnnaMikla changed the task status from Open to In Progress.Oct 26 2022, 11:08 AM

Anribolon updated the task description. (Show Details)Oct 27 2022, 10:45 AM

AnnaMikla changed the task status from In Progress to Open.Oct 28 2022, 12:27 PM

AnnaMikla changed the task status from Open to In Progress.

article update - Actions API (GetPage), Actions API (GetPage - date created), REST API (GetPageHTML), ORES, Actions API (GetUser), TextProcessor gRPC
topics: articles, versions -> 48h history
article bulk - Action API (GetPages - bulk max 50 titles), Actions API (GetPages - date created, can we do bulk?), REST API (GetPageHTML - needs concurrency, maybe semaphore?, create GetPagesHTML with concurrency)
topics: articles (compacted)
how do we process the GetPages and GetPagesHTML to build Article object