Page MenuHomePhabricator
Create Task
Maniphest T324482

[Migration] Oozie Migration jobs for Pageviews
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Migrate Oozie pageview hourly job to Airflow

I think we can leave the historical jobs behind as they hopefully should not be needed - can you confirm @Milimetric and @mforns ? -- Confirmed ~~~~

Details

ProjectBranchLines +/-Subject
analytics/refinerymaster+306 -58Migrate pageview_hourly and related jobs
Customize query in gerrit

Related Objects
Search...

Event Timeline

JArguello-WMF created this task.Dec 5 2022, 4:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2022, 4:49 PM
JArguello-WMF renamed this task from [Migration] Ozzie Migration jobs for Pageviews to [Migration] Oozie Migration jobs for Pageviews.Dec 5 2022, 4:56 PM
JArguello-WMF moved this task from To be discussed /To be estimated to To be prioritised on the Data Pipelines board.
JArguello-WMF updated the task description. (Show Details)Dec 19 2022, 4:43 PM
JArguello-WMF set the point value for this task to 5.
mforns added a subscriber: mforns.Dec 19 2022, 4:43 PM

Let's make sure we tripple-check when vetting the data :]

mforns added a comment.Dec 19 2022, 4:54 PM

And take time to validate that the used UDFs work well in Spark's multi-thread context!

JAllemandou updated the task description. (Show Details)Dec 20 2022, 11:04 AM
JAllemandou added a subscriber: Milimetric.
• EChetty moved this task from To be prioritised to Sprint 07 on the Data Pipelines board.Jan 17 2023, 11:48 AM
• EChetty edited projects, added Data Pipelines (Sprint 07); removed Data Pipelines.
Milimetric claimed this task.Jan 30 2023, 6:16 PM

dibs! Yaay :)

Milimetric moved this task from Ready to In Progress on the Data Pipelines (Sprint 07) board.Jan 31 2023, 5:12 PM
• EChetty moved this task from Sprint 07 to Sprint 08 on the Data Pipelines board.Feb 7 2023, 12:26 PM
• EChetty edited projects, added Data Pipelines (Sprint 08); removed Data Pipelines (Sprint 07).
• EChetty moved this task from Ready to In Progress on the Data Pipelines (Sprint 08) board.
Milimetric added a comment.Feb 8 2023, 5:17 PM

I'm working on a merge request for this, testing the jobs (it's going slow 'cause I'm on ops week)

gerritbot added a comment.Feb 9 2023, 1:16 AM

Change 887869 had a related patch set uploaded (by Milimetric; author: Milimetric):

[analytics/refinery@master] Migrate pageview_hourly and related jobs

https://gerrit.wikimedia.org/r/887869

gerritbot added a project: Patch-For-Review.Feb 9 2023, 1:16 AM
Milimetric moved this task from In Progress to In Review on the Data Pipelines (Sprint 08) board.Feb 9 2023, 1:16 AM

I'm putting this in review, but there are three jobs being migrated so I'll send them in separate patches.

Milimetric updated the task description. (Show Details)Feb 14 2023, 5:09 PM
gerritbot added a comment.Feb 15 2023, 12:22 PM

Change 889532 had a related patch set uploaded (by Milimetric; author: Milimetric):

[analytics/refinery@master] Migrate pageview dumps jobs

https://gerrit.wikimedia.org/r/889532

JArguello-WMF edited projects, added Data Pipelines (Sprint 11); removed Data Pipelines (Sprint 08).Feb 20 2023, 5:24 PM
JArguello-WMF moved this task from Ready to In Review on the Data Pipelines (Sprint 11) board.Feb 20 2023, 5:25 PM
gerritbot added a comment.Fri, Mar 10, 7:24 PM

Change 887869 merged by Milimetric:

[analytics/refinery@master] Migrate pageview_hourly and related jobs

https://gerrit.wikimedia.org/r/887869

JArguello-WMF edited projects, added Data Pipelines (sprint 10); removed Data Pipelines (Sprint 11).Mon, Mar 13, 1:57 PM
JArguello-WMF moved this task from Ready to In Review on the Data Pipelines (sprint 10) board.
JArguello-WMF moved this task from In Review to Done on the Data Pipelines (sprint 10) board.Mon, Mar 13, 4:04 PM
JArguello-WMF closed this task as Resolved.Mon, Mar 20, 6:11 PM
JArguello-WMF closed subtask T329646: [Migration] Oozie Migration jobs for Pageviews dumps as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL