Secret management on airflow for the automated transfer of (public) datasets from stats infra --> WME AWS
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Htriedman
	Wed, Jan 21, 4:48 PM

Description

I've been working with @Snwachukwu to productionize the WME pageviews dataset (airflow-dags, underlying repo). We've merged and gotten it running (see T409601).

Last thing to do here before we have a v0.1 product is upload data from WMF infra --> WME AWS infra.

This should be relatively easy (one shell command added to the airflow DAG), but will require secret management on airflow. Specifically, we need to be able to store some kind of auth token for a user on AWS and include it in the airflow env with this shell command.

Related Objects

Mentioned Here: T409601: Review and productionize the WME differential privacy data set

Event Timeline

Htriedman created this task.Wed, Jan 21, 4:48 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Jan 21, 4:48 PM

Aklapper added a project: Data-Engineering.Wed, Jan 21, 7:03 PM

Ahoelzl moved this task from Incoming (new tickets) to Q3 FY25/26 January 1st - March 31th on the Data-Engineering board.Thu, Jan 22, 5:19 PM

Ahoelzl edited projects, added Data-Engineering (Q3 FY25/26 January 1st - March 31th); removed Data-Engineering.

Secret management on airflow for the automated transfer of (public) datasets from stats infra --> WME AWSOpen, Needs TriagePublicActions

Description

Related Objects

Event Timeline

Secret management on airflow for the automated transfer of (public) datasets from stats infra --> WME AWS
Open, Needs TriagePublic
Actions