Production Airflow dags should be moved to the shared repo
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	gmodena
	Nov 16 2021, 7:37 PM

Description

User Story

As a platform engineer I need to deploy dags to production infra, using generally available tools and practices

Success Criteria

Production platform dags are available at https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags
A scap config is in place to deploy dags and their dependencies to production platform hosts
Our internal tooling is repurposed to target new development hosts

Notes for grooming
Review https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow#Airflow_setup_and_conventions

Related Objects

Mentioned In: T295812: [NEEDS GROOMING] Data pipelines should be published to archiva

Event Timeline

gmodena created this task.Nov 16 2021, 7:37 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2021, 7:37 PM

Let me know when you are ready and I can do the scap part for you.

gmodena mentioned this in T295812: [NEEDS GROOMING] Data pipelines should be published to archiva.Nov 16 2021, 7:43 PM

We're currently working on the dependency management script
that will sync dependencies from archiva (and GitLab?) to HDFS.
I guess this is a blocker for this task, so I'll ping you ASAP when it's done.
Also, from this effort we'll come up with a format for specifying dependencies in a config file.
I'll loop you guys in in the corresponding code reviews, so you can give your thoughts on that.

gmodena added a project: Data Pipelines.Mar 1 2022, 8:23 AM

lbowmaker removed a project: Generated Data Platform.Aug 26 2022, 2:16 PM

JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 5:54 PM

lbowmaker edited projects, added Data-Engineering; removed Data Engineering and Event Platform Team.Nov 10 2023, 2:50 PM

lbowmaker closed this task as Resolved.Feb 12 2024, 2:19 PM

lbowmaker claimed this task.