In order to start creating Airflow jobs for Movement Insights, I need to study Airflow, ask any high-level questions, and do any basic set-up (like creating an team instance if that's what Data Engineering would prefer).
Description
Description
Details
Details
Related Changes in GitLab:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Configure team-specific alert emails | repos/data-engineering/airflow-dags!664 | nshahquinn-wmf | analytics_product_config | main | |
| Revert "Create team-specific EasyDAG templates" | repos/data-engineering/airflow-dags!653 | nshahquinn-wmf | work/nshahquinn-wmf/revert_split_config | main | |
| Create team-specific EasyDAG templates | repos/data-engineering/airflow-dags!651 | nshahquinn-wmf | work/nshahquinn-wmf/split_analytics_product_config | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | nshahquinn-wmf | T359207 Improve the delivery of the movement movements (SDS 2.6.2) | |||
| Invalid | nshahquinn-wmf | T359697 Document tables managed by the movement_metrics ETL job | |||
| Resolved | Mayakp.wiki | T359700 Retire the movement metrics Superset dashboards | |||
| Resolved | nshahquinn-wmf | T333225 Migrate the movement_metrics ETL jobs to Airflow | |||
| Open | None | T359689 Migrate the movement-metrics notebooks to Airflow | |||
| Resolved | nshahquinn-wmf | T359696 Study Airflow and do any necessary set-up | |||
| Resolved | brennen | T360956 Create new GitLab project group: movement-insights |
Event Timeline
Comment Actions
The last thing I need to do I make arrangements to share the analytics_product Airflow instance while still having team-specific configuration (like the alert email address).
Comment Actions
nshahquinn-wmf opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/651
Create team-specific EasyDAG templates
Comment Actions
nshahquinn-wmf merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/651
Create team-specific EasyDAG templates
Comment Actions
nshahquinn-wmf opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/653
Revert "Create team-specific EasyDAG templates"
Comment Actions
bearloga merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/653
Revert "Create team-specific EasyDAG templates"
Comment Actions
In addition to re-doing the configuration change, @Hghani and I need to get permission to access the deployment server so we can deploy jobs.
Comment Actions
nshahquinn-wmf opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/664
Configure team-specific alert emails
Comment Actions
nshahquinn-wmf merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/664
Configure team-specific alert emails