Create an RFC for chunked snapshots
Closed, ResolvedPublic8 Estimated Story Points
Actions

Assigned To

Authored By

	prabhat
	Dec 21 2023, 2:48 PM

Description

O6 KR2
In order to produce and serve chunks of larger snapshots, we need to scope and design this work.
Please start with under progress RfC-chuncked-snapshots.

To do

Finalize API design for chunks metadata and snapshots metadata
Finalize schema design for snaphshot and chunks metadata
Infrastructure - s3 folder/filename key for chunk metadata and chunk itself
Implementation - how to intercept snapshot handler, how to know what snapshots to chunk, number of articles per chunks, etc.

Acceptance criteria
Complete RfC presented to the team

Related Objects
Search...

Status	Assigned	Task
Open	None	T364904 [Epic] O4 KR1 Chunking snapshots
Resolved	ROdonnell-WMF	T353881 Create an RFC for chunked snapshots
Open	None	T355443 Update snapshot schema and build chunks endpoint
Open	None	T364912 Update snapshots service to produce and upload chunks of snapshots
Open	None	T364915 Production deploy snapshot chunks feature
Open	None	T364919 Add integration tests for chunks
Open	None	T364922 Update SDK with chunk feature support

Event Timeline

prabhat created this task.Dec 21 2023, 2:48 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 21 2023, 2:48 PM

JArguello-WMF assigned this task to ROdonnell-WMF.Dec 21 2023, 2:51 PM

JArguello-WMF triaged this task as High priority.

JArguello-WMF set the point value for this task to 8.

JArguello-WMF edited projects, added Wikimedia Enterprise (sprint 53); removed Wikimedia Enterprise.

prabhat updated the task description. (Show Details)Dec 21 2023, 8:14 PM

prabhat updated the task description. (Show Details)

ROdonnell-WMF updated the task description. (Show Details)Jan 2 2024, 2:43 AM

I've written up an RFC proposal that would move away from JSON dumps. Instead I proposed Parquet files and using Kafka Connect to configure the export process.
We'd keep the existing JSON dumps for compatibility and the new format would add a richer client experience and better API usability

More in RFC folder in Implementing Apache Parquet for WME Dumps

Looking for team feedback on RFC titled, Implementing Apache Parquet for WME Dumps

Looked into the RFC for Parquet format. It seems totally unrelated to the chunked snapshots. We should not involve or discuss Parquet format as part of this scoping work. This task is specifically for chunking < 1% of our current larger snapshots.

ROdonnell-WMF moved this task from QA to In Progress on the Wikimedia Enterprise (sprint 53) board.Jan 7 2024, 2:07 PM

Cpetrillo added a project: WME-API-Usability.Jan 8 2024, 1:26 PM

ROdonnell-WMF updated the task description. (Show Details)Jan 9 2024, 4:53 PM

ROdonnell-WMF moved this task from In Progress to Blocked/Paused on the Wikimedia Enterprise (sprint 53) board.Jan 17 2024, 3:08 PM

Based on Phrabat's draft RFC, I'm reservations about the proposed complexity that will be in this one file for export.go. The code is already complex and hard to maintain. The proposed changes would be conditions for turning on/off chunking based on file size and on the export type. I'd prefer to see a re-architecture of this block of code so it is more maintainable and less cryptic code.

I'd like to discuss it with Phrabat to see if he has insights into making the code more maintainable. I've put it in the blocked column until I've chatted with Phrabat.

ROdonnell-WMF moved this task from Blocked/Paused to In Progress on the Wikimedia Enterprise (sprint 53) board.Jan 17 2024, 4:07 PM

JArguello-WMF updated the task description. (Show Details)Jan 18 2024, 2:17 PM

JArguello-WMF edited projects, added Wikimedia Enterprise (Sprint 54); removed Wikimedia Enterprise (sprint 53).Jan 18 2024, 2:28 PM

JArguello-WMF moved this task from Next Up to In Progress on the Wikimedia Enterprise (Sprint 54) board.

ROdonnell-WMF updated the task description. (Show Details)Jan 19 2024, 12:56 AM

prabhat added a subtask: T355443: Update snapshot schema and build chunks endpoint .Jan 19 2024, 5:52 PM

ROdonnell-WMF moved this task from In Progress to QA on the Wikimedia Enterprise (Sprint 54) board.Jan 22 2024, 12:11 PM

JArguello-WMF moved this task from QA to Sign Off on the Wikimedia Enterprise (Sprint 54) board.Jan 22 2024, 2:39 PM

JArguello-WMF moved this task from Sign Off to Done on the Wikimedia Enterprise (Sprint 54) board.Jan 24 2024, 2:08 PM

JArguello-WMF changed the task status from In Progress to Open.Feb 26 2024, 2:34 PM

JArguello-WMF closed this task as Resolved.Mar 1 2024, 12:42 PM

prabhat added a parent task: T364904: [Epic] O4 KR1 Chunking snapshots.Tue, May 14, 6:51 PM