Page MenuHomePhabricator
Create Task
Maniphest T309718

[Airflow] Migrate Oozie's mediawiki_history_load jobs to Airflow
Closed, ResolvedPublic
Actions

Description

Migrate all mediawiki_history_load jobs to Airflow.

  • All jobs live under refinery/oozie/mediawiki/history/load.
  • They are responsible to repair each one of the mediawiki tables (adding the missing partition meta-data) after they have been imported from mariaDB by sqoop. Basically they execute MSCK REPAIR TABLE <tablename> for each table.
  • The job also creates a success file named _PARTITIONED inside each table's partition directory, so that subsequent jobs know when the Hive partition meta-data is complete.
  • Probably it is possible to create just 1 Airflow DAG that iterates over a list of datasets and creates the corresponding tasks. Likely, for each dataset: 1) A sensor 2) The repair table (SparkSQLOperator?) and 3) A URLTouchOperator that generates the _PARTITIONED flag.

Details

SubjectRepoBranchLines +/-
Removed hive.mapred.mode = nonstrict Updated the usage from hive to spark2-sql Added repair_partitions.hql file to hql pathanalytics/refinerymaster+11 -0
Customize query in gerrit

Event Timeline

mforns created this task.Jun 1 2022, 2:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 1 2022, 2:40 PM
• NOkafor-WMF claimed this task.Jun 1 2022, 3:58 PM
mforns added a project: Data-Engineering-Kanban.Jun 6 2022, 3:17 PM
Milimetric moved this task from Next Up to In Code Review on the Data-Engineering-Kanban board.Jun 6 2022, 3:17 PM
Milimetric moved this task from In Code Review to In Progress on the Data-Engineering-Kanban board.
• EChetty moved this task from Backlog to Discussed (Radar) on the Data Pipelines board.Jun 6 2022, 3:39 PM
mforns moved this task from Discussed (Radar) to Estimated on the Data Pipelines board.Jun 6 2022, 3:40 PM
JArguello-WMF removed a project: Data-Engineering.Jun 16 2022, 3:04 PM
gerritbot added a comment.Jun 27 2022, 12:57 PM

Change 808888 had a related patch set uploaded (by NOkafor; author: NOkafor):

[analytics/refinery@master] Added repair_partitions.hql file to hql path

https://gerrit.wikimedia.org/r/808888

gerritbot added a project: Patch-For-Review.Jun 27 2022, 12:57 PM
• EChetty moved this task from Estimated to In Review on the Data Pipelines board.Jun 28 2022, 6:13 PM
JArguello-WMF moved this task from In Progress to In Code Review on the Data-Engineering-Kanban board.Jun 28 2022, 6:15 PM
gerritbot added a comment.Jun 28 2022, 8:28 PM

Change 808888 merged by Mforns:

[analytics/refinery@master] Removed hive.mapred.mode = nonstrict Updated the usage from hive to spark2-sql Added repair_partitions.hql file to hql path

https://gerrit.wikimedia.org/r/808888

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2022, 8:30 PM
• NOkafor-WMF moved this task from In Review to Done on the Data Pipelines board.Jun 30 2022, 10:36 AM
• NOkafor-WMF moved this task from In Code Review to Done on the Data-Engineering-Kanban board.
JArguello-WMF closed this task as Resolved.Jul 5 2022, 3:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL