User Story

As a Data Engineer, I need to migrate dataset: XXX to Apache Iceberg (in planning we suggested pageview_hourly or unique devices datasets to begin with)

Why?

So that the team can understand how to do the migration using a small dataset

Success Criteria

• Dataset: xxx is migrated to Iceberg
• Documented in DataHub (including how to query them)

Technical details:

We plan on migrating a relatively small dataset (or dataset - set), like unique-devices, projectview, referer or similar.
Migrating the dataset to iceberg is not very complicated, it entails adding the timestamp to the data if it's not already present (or present in the partition only) and reinserting old data into a newly defined iceberg table.

Things still to investigate:

• How to make Airlflow sensors to react to new Iceberg data (the hive partition sensor won't work)
  • Jobs to compact / maintain Iceberg tables - there will be many of those, we should think about how we wish to build for that (one job with multiple datasets to process from config, or one job per dataset to process).