Harmonize tags across Airflow dags
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	Antoine_Quhen
	May 16 2023, 10:06 AM

Description

If you search for druid in the Airflow UI, you may not get the whole list of dags interacting with Druid.

We should be able to automatically add tags to a dag when we add a specific kind of operator. e.g., using the HiveToDruidOperator => adding the druid tag

And generally speaking, we should review the tags added to all the dags we have created.

DE 30 min meeting
Implementation
Publish criteria

Details

	Title	Reference	Author	Source Branch	Dest Branch
	Harmonize Tags on Airflow	repos/data-engineering/airflow-dags!460	jebe	T336744-Harmonize-tags-in-Airflow-dags	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T336739 Post Oozie -> Airflow migration refactorings
		Resolved		JEbe-WMF	T336744 Harmonize tags across Airflow dags

Event Timeline

Antoine_Quhen created this task.May 16 2023, 10:06 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 16 2023, 10:06 AM

Antoine_Quhen added a parent task: T336739: Post Oozie -> Airflow migration refactorings.May 16 2023, 10:06 AM

lbowmaker moved this task from Backlog to To be discussed /To be estimated on the Data Pipelines board.May 30 2023, 1:31 PM

+1 to revise all tags manually.

Not sure whether automatizing this will add value, as the manual review should take an hour?

Today we decided not to automatize the tagging process.

JArguello-WMF edited projects, added Data Pipelines (Sprint 14); removed Data Pipelines.Jun 5 2023, 4:40 PM

JArguello-WMF updated the task description. (Show Details)

JArguello-WMF set the point value for this task to 3.

mforns claimed this task.Jun 22 2023, 3:37 PM

mforns moved this task from Next Up to In Progress on the Data Pipelines (Sprint 14) board.

mforns reassigned this task from mforns to JEbe-WMF.Jun 22 2023, 4:14 PM

mforns subscribed.

Here's a list of all Airflow analytics tags as of 2023-06-22.
https://docs.google.com/spreadsheets/d/1XtvtLeZUWIWmEYGF9JYukeszZZ1oO0je8FJVbOKyxpA

The tag types we have so far include: data destination, type of data generated, dataset family, technology used, update schedule, job purpose, DAG property.

Some things I noticed that we could improve (subjective, of course!):

Lots of tags are used only once, usually referring to a small dataset family (i.e. referrer, metadata, virtualpageview). I think these are not useful, since they do not group DAGs into categories. They don't add much information.
Some tags are used in many DAGs, and could possibly be used in almost all DAGs (i.e. hive, hql, spark). I think these are not useful either, since they don't filter out enough DAGs. Also not much information added.
Some tags are vague (i.e. alert, ingest, load, email, check). Most DAGs do load data and ingest in some way, they also alert and send emails in case of failure. Also not much info added.
Some tags are duplicated (i.e. druid and druid_load, history and mediawiki_history, dump and dumps). We should unify those, no?
Some tags give information that is already in the DAGs summary in Airflow Home (i.e. hourly, daily, monthly, yearly). I don't think we should duplicate that info?

We did a 'functional` decomposition of current datasets available under the wmf database in Hive over at T337562: Decide how to split wmf database into functional areas. I think the categorization we got done there could help here as well:

contributors
data_ops
experiments
mediawiki
readership
traffic
wikidata

You can see rationale for this at T337562, and the final output available at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Iceberg#Changes_to_database_names.

The data engineering team had a meeting and the conclusion was capture tags based on
*Frequency,
*Ownership,
*Criticality,
*Requires a certain table e ie Webrequest
*Destination of data source ie Iceberg, Hive