Maniphest T311743

[Iceberg] Epic: Icebergify event_sanitized database
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Ottomata
	Jun 30 2022, 3:54 PM

Tags

Referenced Files

None

Subscribers

Description

Parent task for ongoing work to create an event_sanitized database using iceberg tables.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T311743 [Iceberg] Epic: Icebergify event_sanitized database
		Open		None	T311737 [Iceberg] Migrate event_sanitized_iceberg to event_sanitized
		Open		None	T225751 Consider renaming event and event_sanitized Hive databases
		Duplicate		None	T236794 Find a strategy to mitigate small-files handling for long-term kept events
		Open		JAllemandou	T311739 [Iceberg] Update Refine Sanitize to insert into Iceberg tables
		Resolved		xcollazo	T311738 [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive
		Resolved		Ottomata	T311525 Upgrade to latest PrestoDB and enable iceberg support
		Declined		Stevemunene	T324011 SPIKE: Spin up a Test Trino instance (Evaluate Trino)
		Resolved		xcollazo	T335721 Add support for Iceberg in Spark
		Open		xcollazo	T336012 Add support for Iceberg to the Spark Docker Image

Event Timeline

Ottomata created this task.Jun 30 2022, 3:54 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2022, 3:54 PM

Ottomata added subtasks: T311737: [Iceberg] Migrate event_sanitized_iceberg to event_sanitized, T311739: [Iceberg] Update Refine Sanitize to insert into Iceberg tables, T311738: [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive.Jun 30 2022, 3:55 PM

• EChetty added a project: Epic.Jun 30 2022, 3:59 PM

• EChetty moved this task from Backlog to Epics on the Data-Engineering-Planning board.Jun 30 2022, 5:47 PM

gmodena subscribed.Jul 7 2022, 1:47 PM

xcollazo subscribed.May 1 2023, 1:58 PM

xcollazo changed the status of subtask T311738: [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive from Open to In Progress.May 2 2023, 12:47 AM

xcollazo closed subtask T311738: [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive as Resolved.Jun 2 2023, 7:52 PM

JArguello-WMF moved this task from Epics to Pipelines on the Data-Engineering-Planning board.Jun 29 2023, 9:42 PM

JArguello-WMF added a project: Data Pipelines.

JArguello-WMF removed a project: Data-Engineering-Planning.Jun 29 2023, 9:59 PM

JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 5:53 PM

lbowmaker edited projects, added Data-Engineering; removed Data Engineering and Event Platform Team.Nov 10 2023, 2:49 PM

Relevant slack discussion: https://app.slack.com/client/E012JBDTTHA/CSV483812

We could take advantage of this migration to delete some unused data and implement a smaller default retention policy than all-time.

Ottomata merged a task: T236794: Find a strategy to mitigate small-files handling for long-term kept events.Dec 14 2023, 2:22 PM

Ottomata added a subscriber: • Tbayer.

lbowmaker moved this task from Incoming (new tickets) to Icebox (not considered in current quarter) on the Data-Engineering board.Feb 14 2024, 3:46 PM