Page MenuHomePhabricator

[Analytics] [Tech Debt] Use `iceberg_` instead of `hive_` for Airflow prop identifiers for `wmde` tables
Closed, ResolvedPublic

Description

Wikidata Analytics Request

This task was generated using the Wikidata Analytics request form. Please use the task templates linked on our project page to create tasks for the team. Thank you!

Purpose

Please provide as much context as possible as well as what the produced insights or services will be used for.

This is purely a variable rename change. I'm realizing that I was using hive_ in prop names that reference tables in wmde Airflow instance DAGs, but all wmde schema tables are Iceberg tables, so we should instead use iceberg_ as the prefix as this appears to be how WMF does it.

Desired Outputs

The desired outputs of this task are listed as check boxes and confirmed as being finished below.

  • Replace all instances of hive_ to iceberg_ for wmde DAGs when referencing wmde instance tables
  • Redeploy all WMDE DAGs

Deadline

Please make the time sensitivity of this request clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority.

Low priority tech debt


Information below this point is filled out by the task assignee.

Assignee Planning

Sub Tasks

A full breakdown of the steps to complete this task.

See desired outputs above

Estimation

Estimate: 1 hour
Actual: 1 hour (including testing)

Notes

Things that came up during the completion of this task, questions to be answered and follow up tasks.

  • Note

Event Timeline

andrewtavis-wmde merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1970

T415908 Update DAG prop ids to indicate that WMDE tables are iceberg, not hive

AndrewTavis_WMDE updated the task description. (Show Details)
AndrewTavis_WMDE added a subscriber: karapayneWMDE.

Resolving based on the work in !1970 ✅ CC @karapayneWMDE :)