Page MenuHomePhabricator
Create Task
Maniphest T285692

Write a job entirely in Airflow with spark and/or sparkSQL
Closed, ResolvedPublic
Actions

Description

We should write and test a full Airflow DAG that mirrors a job we have in production.
It should have a SparkSQL (converted from Hive) query and a Spark job.
If we need, we should have 2 jobs that test both of these connectors.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add airflow DAG for anomaly detection (POC)analytics/refinerymaster+502 -0
Simplify RSVD anomaly detection job for Airflow POCanalytics/refinery/sourcemaster+60 -85
Move airflow-analytics-test instance to an-test-client1001operations/puppetproduction+39 -1
airflow - set default smtp settingsoperations/puppetproduction+13 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

mforns created this task.Jun 28 2021, 3:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 28 2021, 3:47 PM
mforns triaged this task as High priority.Jun 28 2021, 3:47 PM
mforns moved this task from Incoming to Airflow on the Analytics board.
mforns updated the task description. (Show Details)
mforns added a parent task: T284172: [SPIKE] analytics-airflow jobs development.Jun 28 2021, 3:52 PM
mforns added a project: Analytics-Kanban.Jun 29 2021, 1:49 PM
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot added a comment.Jul 1 2021, 2:20 PM

Change 702668 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery@master] Add airflow DAG for anomaly detection (POC)

https://gerrit.wikimedia.org/r/702668

gerritbot added a project: Patch-For-Review.Jul 1 2021, 2:20 PM
gerritbot added a comment.Jul 23 2021, 6:15 PM

Change 707489 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] airflow - set default smtp settings

https://gerrit.wikimedia.org/r/707489

gerritbot added a comment.Jul 23 2021, 6:22 PM

Change 707489 merged by Ottomata:

[operations/puppet@production] airflow - set default smtp settings

https://gerrit.wikimedia.org/r/707489

gerritbot added a comment.Jul 23 2021, 6:57 PM

Change 707517 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery/source@master] Simplify RSVD anomaly detection job for Airflow POC

https://gerrit.wikimedia.org/r/707517

gerritbot added a comment.Jul 27 2021, 3:49 PM

Change 708314 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Move airflow-analytics-test instance to an-test-client1001

https://gerrit.wikimedia.org/r/708314

gerritbot added a comment.Jul 27 2021, 3:54 PM

Change 708314 merged by Ottomata:

[operations/puppet@production] Move airflow-analytics-test instance to an-test-client1001

https://gerrit.wikimedia.org/r/708314

Ottomata merged a task: T284172: [SPIKE] analytics-airflow jobs development.Aug 3 2021, 4:08 PM
Ottomata added subscribers: Ottomata, EBernhardson, fkaelin and 6 others.
mforns moved this task from In Progress to Done on the Analytics-Kanban board.Sep 17 2021, 4:26 PM
• odimitrijevic added projects: Data-Engineering-Kanban, Data-Engineering.Oct 28 2021, 5:11 AM
• odimitrijevic moved this task from Next Up to Done on the Data-Engineering-Kanban board.Oct 28 2021, 5:15 AM
• odimitrijevic moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.Oct 28 2021, 5:56 AM
mforns edited projects, added Data Pipelines; removed Analytics-Kanban, Analytics.Nov 5 2021, 3:43 PM
mforns moved this task from Backlog to Done on the Data Pipelines board.Nov 5 2021, 8:17 PM
gerritbot added a comment.Dec 14 2021, 5:14 PM

Change 707517 merged by jenkins-bot:

[analytics/refinery/source@master] Simplify RSVD anomaly detection job for Airflow POC

https://gerrit.wikimedia.org/r/707517

gerritbot added a comment.Jan 20 2022, 2:20 PM

Change 702668 abandoned by Mforns:

[analytics/refinery@master] Add airflow DAG for anomaly detection (POC)

Reason:

This code has been migrated to https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags

https://gerrit.wikimedia.org/r/702668

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2022, 3:10 PM
JArguello-WMF closed this task as Resolved.May 31 2022, 3:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits