Low Risk Oozie Migration: 4 wikidata metrics jobs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• EChetty
	Jan 25 2022, 11:39 AM

Description

Goal:
Migrate the 3 wikidata metrics oozie jobs to Airflow:

Job Details:

Input	Processing	Output
Hive Table	Spark	Graphite

Success Criteria:

Have the 3 daily Jobs Migrated (SLA 30 Hours) - Review if jobs could be merged for performance improvement
Have the 1 Monthly Job Migrated (SLA 40 days)

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Update wikidata metrics hql script	analytics/refinery	master	+2 -3
	Add HQL scripts for wikidata graphite metrics	analytics/refinery	master	+296 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• odimitrijevic	T282033 Airflow collaborations
Resolved	• odimitrijevic	T271429 Replace Oozie with better workflow scheduler
Resolved	• odimitrijevic	T299074 Migrate Oozie jobs to Airflow
Resolved	Snwachukwu	T300021 Low Risk Oozie Migration: 4 wikidata metrics jobs
Resolved	Snwachukwu	T306177 Low RIsk Ozzie Migration: Wikidata CoEditor metric job

Event Timeline

• EChetty created this task.Jan 25 2022, 11:39 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2022, 11:39 AM

• EChetty updated the task description. (Show Details)Jan 25 2022, 11:42 AM

• EChetty added a parent task: T299074: Migrate Oozie jobs to Airflow.Jan 25 2022, 12:01 PM

JAllemandou renamed this task from Low Risk Ozzie Migration: wikidata _metrics to Low Risk Ozzie Migration: 4 wikidata metrics jobs.Jan 25 2022, 1:16 PM

JAllemandou updated the task description. (Show Details)

• odimitrijevic moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.Feb 1 2022, 11:35 PM

• EChetty moved this task from Backlog to Discussed (Radar) on the Data Pipelines board.Feb 8 2022, 11:43 AM

Snwachukwu claimed this task.Feb 28 2022, 4:14 PM

• EChetty moved this task from Discussed (Radar) to Estimated on the Data Pipelines board.Feb 28 2022, 4:16 PM

• EChetty moved this task from Estimated to Discussed (Radar) on the Data Pipelines board.Feb 28 2022, 4:44 PM

• EChetty moved this task from Discussed (Radar) to Estimated on the Data Pipelines board.Mar 7 2022, 4:34 PM

Snwachukwu moved this task from Estimated to Discussed (Radar) on the Data Pipelines board.Mar 14 2022, 2:52 PM

• EChetty moved this task from Discussed (Radar) to Estimated on the Data Pipelines board.Mar 21 2022, 3:56 PM

Snwachukwu moved this task from Estimated to Discussed (Radar) on the Data Pipelines board.Apr 14 2022, 12:27 PM

Snwachukwu moved this task from Discussed (Radar) to Estimated on the Data Pipelines board.May 11 2022, 6:03 PM

Snwachukwu added a project: Data-Engineering-Kanban.May 11 2022, 6:21 PM

Snwachukwu moved this task from Next Up to In Progress on the Data-Engineering-Kanban board.

Change 791394 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):

[analytics/refinery@master] Create HQL scripts to generate Wikidata's ArticlePlaceholder and Reliability metrics.

https://gerrit.wikimedia.org/r/791394

gerritbot added a project: Patch-For-Review.May 12 2022, 3:24 PM

• EChetty moved this task from Estimated to In Review on the Data Pipelines board.May 23 2022, 3:38 PM

JArguello-WMF moved this task from In Progress to In Code Review on the Data-Engineering-Kanban board.May 25 2022, 4:16 PM

JArguello-WMF closed subtask T306177: Low RIsk Ozzie Migration: Wikidata CoEditor metric job as Resolved.May 31 2022, 3:25 PM

Change 791394 merged by Mforns:

[analytics/refinery@master] Add HQL scripts for wikidata graphite metrics

https://gerrit.wikimedia.org/r/791394

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2022, 6:30 PM

Milimetric moved this task from In Code Review to Ready to Deploy on the Data-Engineering-Kanban board.Jun 6 2022, 3:06 PM

Milimetric moved this task from Ready to Deploy to In Code Review on the Data-Engineering-Kanban board.

Change 803306 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):

[analytics/refinery@master] Update wikidata metrics hql script

https://gerrit.wikimedia.org/r/803306

gerritbot added a project: Patch-For-Review.Jun 6 2022, 3:18 PM

Change 803306 merged by Mforns:

[analytics/refinery@master] Update wikidata metrics hql script

https://gerrit.wikimedia.org/r/803306

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2022, 4:30 PM

Ottomata renamed this task from Low Risk Ozzie Migration: 4 wikidata metrics jobs to Low Risk Oozie Migration: 4 wikidata metrics jobs.Jun 7 2022, 4:09 PM

Snwachukwu moved this task from In Code Review to Ready to Deploy on the Data-Engineering-Kanban board.Jun 20 2022, 6:12 AM

Snwachukwu moved this task from Ready to Deploy to Done on the Data-Engineering-Kanban board.Jun 21 2022, 2:59 PM

Snwachukwu moved this task from In Review to Done on the Data Pipelines board.

Michael subscribed.Jun 22 2022, 1:15 PM

Michael mentioned this in T304793: Get metrics on usage of Entity Schemas.Jun 22 2022, 2:09 PM

JArguello-WMF closed this task as Resolved.Jul 5 2022, 3:52 PM

Michael mentioned this in T314131: Some reliability metrics missing since June 20th '22.Jul 29 2022, 11:23 AM