[Iceberg Migration] Migrate unique devices tables to Iceberg
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• lbowmaker
	Sep 29 2023, 12:55 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

• lbowmaker

xcollazo

Description

Migrate unique devices tables to Iceberg format.

https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Unique_Devices

See this ticket for details of the referrer_daily migration https://phabricator.wikimedia.org/T335305

Success Criteria

Update Wikitech page with any useful steps to assist other migrations
Unique devices tables are migrated to Iceberg format (Hive tables will remain for now - there will be a separate ticket to deprecate that will require coordination with consumers)
Update migration status here

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Add unique-devices Iceberg schemas and scripts	Joal	analytics/refinery	master	+425 -0

Customize query in gerrit

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
Update analytics datahub ingestion config	repos/data-engineering/airflow-dags!528	joal	add_analytics_datahub_hive_wmf_readership	main
Fix typo in Iceberg tables database name	repos/data-engineering/airflow-dags!521	aqu	T347689_fix_typo	main
Update analytics unique-devices dags	repos/data-engineering/airflow-dags!515	joal	add_unique_devices_iceberg	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Open	None	T333013 [Iceberg Migration] Apache Iceberg Migration
Resolved	JAllemandou	T347689 [Iceberg Migration] Migrate unique devices tables to Iceberg
Resolved	nshahquinn-wmf	T349132 Update unique devices table in Movement Insights-owned code

Event Timeline

• lbowmaker created this task.Sep 29 2023, 12:55 PM

• lbowmaker moved this task from Data Eng Backlog to Apache Iceberg Migration on the Data Engineering and Event Platform Team board.

• lbowmaker moved this task from Apache Iceberg Migration to Sprint 3 on the Data Engineering and Event Platform Team board.Oct 2 2023, 2:41 PM

• lbowmaker edited projects, added: Data Engineering and Event Platform Team (Sprint 3); removed: Data Engineering and Event Platform Team.

• lbowmaker assigned this task to JAllemandou.Oct 2 2023, 2:48 PM

• lbowmaker mentioned this in T347879: [Airflow Migration] Migrate Airflow Druid Jobs to Unique Devices Iceberg tables.Oct 2 2023, 3:18 PM

• lbowmaker updated the task description. (Show Details)Oct 3 2023, 10:31 AM

JAllemandou moved this task from Next Up to In progress on the Data Engineering and Event Platform Team (Sprint 3) board.Oct 3 2023, 11:45 AM

JAllemandou updated the task description. (Show Details)Oct 3 2023, 1:51 PM

JAllemandou moved this task from In progress to In Review on the Data Engineering and Event Platform Team (Sprint 3) board.Oct 10 2023, 3:49 PM

Change 964939 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery@master] Add unique-devices Iceberg schemas and scripts

https://gerrit.wikimedia.org/r/964939

gerritbot added a project: Patch-For-Review.Oct 10 2023, 7:19 PM

xcollazo subscribed.Oct 12 2023, 3:01 PM

joal opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/515

Update analytics unique-devices dags

Change 964939 merged by TChin:

[analytics/refinery@master] Add unique-devices Iceberg schemas and scripts

https://gerrit.wikimedia.org/r/964939

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/515

Update analytics unique-devices dags

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2023, 2:10 PM

JAllemandou moved this task from In Review to Done on the Data Engineering and Event Platform Team (Sprint 3) board.Oct 17 2023, 4:58 PM

JAllemandou updated the task description. (Show Details)

Mayakp.wiki mentioned this in T349132: Update unique devices table in Movement Insights-owned code.Oct 17 2023, 6:48 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/521

Fix typo in Iceberg tables database name

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/521

Fix typo in Iceberg tables database name

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 9:10 AM

Ahoelzl renamed this task from Migrate unique devices tables to Iceberg to [Iceberg Migration] Migrate unique devices tables to Iceberg.Oct 20 2023, 4:52 PM

Ahoelzl closed this task as Resolved.Oct 24 2023, 2:18 AM

joal opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/528

Update analytics datahub ingestion config

nshahquinn-wmf closed subtask T349132: Update unique devices table in Movement Insights-owned code as Resolved.Oct 25 2023, 6:18 PM

joal merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/528

Update analytics datahub ingestion config

Maintenance_bot removed a project: Patch-For-Review.Nov 15 2023, 1:30 PM

joal opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/545

Update analytics datahub ingestion config