Page MenuHomePhabricator
Create Task
Maniphest T333013

[Iceberg Migration] Apache Iceberg Migration
Open, Needs TriagePublic
Actions

Description

User Story
As a Data Engineer I need to migrate datasets to Apache Iceberg
Why?

So that we can improve the management of existing data files, improve query performance and facilitate faster production of incremental datasets

Migration status: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Iceberg#Which_tables_are_migrated

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT333013 [Iceberg Migration] Apache Iceberg Migration
 DeclinedNoneT333015 Automate installation of Iceberg on the Analytics Cluster
 DeclinedNoneT334601 [needs updating] Migration Dataset: XXX to Apache Iceberg
 ResolvedxcollazoT335305 Migrate referrer_daily to Iceberg
 ResolvedSpikemfornsT335306 [SPIKE] Evaluation on iceberg sensor for airflow
 OpenNoneT340463 [Iceberg Migration] P.O.C. on Iceberg sensor using Iceberg table to keep status of updates
 OpenNoneT340466 [Iceberg Migration] P.O.C. on Iceberg sensor using Postgres table to keep status of updates
 OpenNoneT340471 [Iceberg Migration] P.O.C. on Iceberg sensor using Snapshot metadata to keep status of updates
 ResolvedxcollazoT311738 [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive
 ResolvedOttomataT311525 Upgrade to latest PrestoDB and enable iceberg support
 DeclinedStevemuneneT324011 SPIKE: Spin up a Test Trino instance (Evaluate Trino)
 ResolvedxcollazoT335721 Add support for Iceberg in Spark
 OpenxcollazoT336012 Add support for Iceberg to the Spark Docker Image
 OpenNoneT336513 Figure out a way to automatize deployment of the spark assembly file
 ResolvedBTullisT337335 Upgrade Presto to release that aligns with Iceberg 1.2.1
 ResolvedBTullisT295589 Upgrade Presto to access UDF library improvements
 ResolvedxcollazoT335314 Make sure new iceberg data can be queried from Presto and Spark as well as non-Iceberg data
 ResolvedxcollazoT337562 Decide how to split wmf database into functional areas
 OpenNoneT337416 [Iceberg] performance test to align row group to HDFS block size
 OpenNoneT338050 Benchmark Iceberg tables with SNAPPY vs GZIP vs ZSTD
 OpenNoneT338057 Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0
 OpenNoneT354733 Create Conda Analytics environment for new Spark version
 OpenNoneT354735 Validating that current jobs are running correctly with the new Conda Analytics environment
 OpenNoneT354737 Deploy new Spark version on production
 OpenNoneT338065 [Iceberg Migration] Implement mechanism for automatic Iceberg data deletion and optimization
 ResolvedJAllemandouT347689 [Iceberg Migration] Migrate unique devices tables to Iceberg
 Resolvednshahquinn-wmfT349132 Update unique devices table in Movement Insights-owned code
 OpenNoneT347690 [Iceberg Migration] Migrate pageview tables to Iceberg
 ResolvedJAllemandouT347693 [Iceberg Migration] Deprecate referrer_daily Hive tables
 ResolvedAntoine_QuhenT347879 [Airflow Migration] Migrate Airflow Druid Jobs to Unique Devices Iceberg tables
 ResolvedtchinT352669 [Iceberg Migration] Migrate aqs hourly tables to Iceberg
 ResolvedSnwachukwuT352670 [Iceberg Migration] Migrate browser_general tables to Iceberg
 ResolvedtchinT352671 [Iceberg Migration] Migrate interlanguage tables to Iceberg
 ResolvedAntoine_QuhenT352672 [Iceberg Migration] Migrate session length tables to Iceberg
 OpenSpikeNoneT360922 [Status Store] [SPIKE] Investigate and document approach for Iceberg Sensors

Event Timeline

lbowmaker created this task.Mar 24 2023, 3:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 24 2023, 3:59 PM
lbowmaker updated the task description. (Show Details)Apr 12 2023, 2:03 PM
lbowmaker moved this task from Epics to To be discussed /To be estimated on the Data Pipelines board.Apr 12 2023, 2:06 PM
lbowmaker moved this task from To be discussed /To be estimated to Epics on the Data Pipelines board.Apr 12 2023, 2:08 PM
lbowmaker closed subtask T334601: [needs updating] Migration Dataset: XXX to Apache Iceberg as Declined.Apr 24 2023, 10:38 PM
lbowmaker added a subtask: T335305: Migrate referrer_daily to Iceberg.
lbowmaker added a subtask: T335306: [SPIKE] Evaluation on iceberg sensor for airflow.
JArguello-WMF closed subtask T333015: Automate installation of Iceberg on the Analytics Cluster as Declined.Apr 25 2023, 2:46 PM
JArguello-WMF added a subtask: T311738: [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive.
xcollazo changed the status of subtask T311738: [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive from Open to In Progress.May 2 2023, 12:47 AM
BTullis mentioned this in T337471: Jupyterhub is broken in conda-analytics 0.0.13.May 25 2023, 12:09 PM
xcollazo added a subtask: T335314: Make sure new iceberg data can be queried from Presto and Spark as well as non-Iceberg data.May 26 2023, 2:23 PM
xcollazo changed the status of subtask T337562: Decide how to split wmf database into functional areas from Open to In Progress.May 26 2023, 2:34 PM
xcollazo mentioned this in T337562: Decide how to split wmf database into functional areas.May 26 2023, 2:40 PM
xcollazo closed subtask T335314: Make sure new iceberg data can be queried from Presto and Spark as well as non-Iceberg data as Resolved.May 30 2023, 7:37 PM
xcollazo added a subtask: T337416: [Iceberg] performance test to align row group to HDFS block size.Jun 2 2023, 4:57 PM
xcollazo closed subtask T311738: [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive as Resolved.Jun 2 2023, 7:52 PM
xcollazo closed subtask T335305: Migrate referrer_daily to Iceberg as Resolved.Jun 2 2023, 8:40 PM
xcollazo closed subtask T337562: Decide how to split wmf database into functional areas as Resolved.Jun 9 2023, 2:49 PM
BTullis closed subtask T337335: Upgrade Presto to release that aligns with Iceberg 1.2.1 as Resolved.Jun 27 2023, 3:58 PM
JArguello-WMF closed subtask T335306: [SPIKE] Evaluation on iceberg sensor for airflow as Resolved.Jun 29 2023, 4:29 PM
JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 5:30 PM
JArguello-WMF moved this task from Data Eng Backlog to Parent Tasks/Epics on the Data Engineering and Event Platform Team board.Jun 30 2023, 5:35 PM
BTullis mentioned this in T344910: Deploy additional yarn shuffler services to support several versions of spark in parallel.Aug 24 2023, 11:57 AM
BTullis closed subtask T338057: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 as Resolved.Aug 24 2023, 11:59 AM
Gehel reopened subtask T338057: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 as Open.Aug 24 2023, 6:33 PM
lbowmaker moved this task from Parent Tasks/Epics to Apache Iceberg Migration on the Data Engineering and Event Platform Team board.Sep 8 2023, 1:08 PM
lbowmaker updated the task description. (Show Details)Sep 29 2023, 12:50 PM
lbowmaker updated the task description. (Show Details)Oct 2 2023, 7:40 PM
Ahoelzl renamed this task from Apache Iceberg Migration to [Iceberg Migration] Apache Iceberg Migration.Oct 20 2023, 5:04 PM
Ahoelzl closed subtask T347693: [Iceberg Migration] Deprecate referrer_daily Hive tables as Resolved.Oct 24 2023, 2:17 AM
Ahoelzl closed subtask T347689: [Iceberg Migration] Migrate unique devices tables to Iceberg as Resolved.
Mayakp.wiki subscribed.Oct 24 2023, 5:56 PM
lbowmaker edited projects, added Data-Engineering; removed Data Pipelines.Nov 10 2023, 1:31 PM
lbowmaker moved this task from Incoming (new tickets) to Apache Iceberg Migration on the Data-Engineering board.Nov 10 2023, 1:31 PM
lbowmaker removed a project: Data Engineering and Event Platform Team.Nov 10 2023, 1:34 PM
tchin changed the status of subtask T352669: [Iceberg Migration] Migrate aqs hourly tables to Iceberg from Open to In Progress.Dec 14 2023, 6:16 AM
Antoine_Quhen closed subtask T347879: [Airflow Migration] Migrate Airflow Druid Jobs to Unique Devices Iceberg tables as Resolved.Jan 25 2024, 2:58 PM
lbowmaker closed subtask T352671: [Iceberg Migration] Migrate interlanguage tables to Iceberg as Resolved.Jan 30 2024, 12:19 PM
Antoine_Quhen changed the status of subtask T352672: [Iceberg Migration] Migrate session length tables to Iceberg from Open to In Progress.Jan 31 2024, 12:53 PM
lbowmaker closed subtask T352672: [Iceberg Migration] Migrate session length tables to Iceberg as Resolved.Feb 16 2024, 3:21 PM
lbowmaker closed subtask T352670: [Iceberg Migration] Migrate browser_general tables to Iceberg as Resolved.
lbowmaker closed subtask T352669: [Iceberg Migration] Migrate aqs hourly tables to Iceberg as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL