Page MenuHomePhabricator

[Iceberg Migration] Migrate browser_general tables to Iceberg
Closed, ResolvedPublic5 Estimated Story Points

Description

Migrate browser_general tables to Iceberg format.

https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Browser_general

Success Criteria

  • browser_general tables are migrated to Iceberg format (Hive tables will remain for now - there will be a separate ticket to deprecate that will require coordination with consumers)

Details

TitleReferenceAuthorSource BranchDest Branch
Update browser_general dag to generate iceberg tablerepos/data-engineering/airflow-dags!576ebysansbrowser_general_icebergmain
Customize query in GitLab

Event Timeline

Change 988711 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):

[analytics/refinery@master] Migration of browser General table to iceberg format. 1. Add a iceberg create table statement hql file fof browser_general table 2. Add hql file to update browser_general iceberg table with values.

https://gerrit.wikimedia.org/r/988711

Update a patch containing 2 hql files required to create and update iceberg version of browser_general tables respectively.

Change 988711 merged by Aqu:

[analytics/refinery@master] Migration of browser General table to iceberg format.

https://gerrit.wikimedia.org/r/988711

Ahoelzl set the point value for this task to 5.Jan 30 2024, 7:13 PM