Maniphest T335859

Document Solutions and Decisions for Realtime Dump Implementation
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	• lbowmaker
	May 3 2023, 1:20 PM

Tags

Referenced Files

None

Subscribers

Description

Document the technical decisions taken for the implementation.

Done is:

Doc or wiki page with technical choices and reasoning

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Milimetric	T330296 Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark
		Resolved		Milimetric	T335859 Document Solutions and Decisions for Realtime Dump Implementation
		Resolved		xcollazo	T335860 Implement job to transform mediawiki.page_content_change
		Duplicate		None	T336718 Write data to Iceberg formatted tables (mediawiki.page_content_change)

Event Timeline

• lbowmaker created this task.May 3 2023, 1:20 PM

• lbowmaker moved this task from Backlog to To be discussed /To be estimated on the Data Pipelines board.

• lbowmaker renamed this task from Document approach to Document Solutions and Decisions for Realtime Dump Implementation.May 3 2023, 1:22 PM

• lbowmaker mentioned this in T330296: Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark.May 3 2023, 1:37 PM

JArguello-WMF edited projects, added Data Pipelines (Sprint 12); removed Data Pipelines.May 15 2023, 4:19 PM

JArguello-WMF set the point value for this task to 3.

• lbowmaker moved this task from Sprint 12 to Sprint 14 on the Data Pipelines board.May 15 2023, 4:19 PM

• lbowmaker edited projects, added Data Pipelines (Sprint 14); removed Data Pipelines (Sprint 12).

Milimetric claimed this task.May 22 2023, 7:09 PM

Milimetric moved this task from Next Up to In Progress on the Data Pipelines (Sprint 14) board.

xcollazo changed the status of subtask T335860: Implement job to transform mediawiki.page_content_change from Open to In Progress.Jun 5 2023, 8:29 PM

Milimetric updated the task description. (Show Details)Jun 12 2023, 3:52 PM

Milimetric moved this task from In Progress to In Review on the Data Pipelines (Sprint 14) board.

Milimetric moved this task from In Review to Done on the Data Pipelines (Sprint 14) board.Jun 20 2023, 4:07 PM

JArguello-WMF closed this task as Resolved.Jun 29 2023, 4:29 PM

xcollazo closed subtask T335860: Implement job to transform mediawiki.page_content_change as Resolved.Jul 25 2023, 3:36 PM