Page MenuHomePhabricator
Create Task
Maniphest T324483

[Migration] Pageview - Learning
Closed, ResolvedPublic9 Estimated Story Points
Actions

Description

Migrate oozie jobs for Pageview - Learning
This encompasses four oozie jobs:

  • oozie/learning/features/actor/hourly
  • oozie/learning/features/actor/rollup/hourly
  • oozie/learning/predictions/actor/hourly
  • oozie/pageview/actor

Also, the learning, features and predictions vocable were used when we were thinking that new "ML" style production jobs would land on the cluster. This has not proven true so far. Do we wish to change those names?

This task needs to be broken down in three:

  • Make UDFs work with Spark (multithreading)
  • Update HQL files for the jobs to make them spark compliant
  • Migrate the job to Airflow using the updated UDFs and HQL files

Details

SubjectRepoBranchLines +/-
Remove previously absent timers from analytics data_purgeoperations/puppetproduction+0 -25
Update analytics data purge for webrequest_actoroperations/puppetproduction+29 -4
Add webrequest and pageview actor scriptsanalytics/refinerymaster+331 -19
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
Add analytics webrequest and pageview actor dagsrepos/data-engineering/airflow-dags!223joalwebrequest_actormain
Customize query in GitLab

Event Timeline

JArguello-WMF created this task.Dec 5 2022, 4:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2022, 4:52 PM
JArguello-WMF moved this task from To be discussed /To be estimated to To be prioritised on the Data Pipelines board.Dec 5 2022, 4:57 PM
JArguello-WMF updated the task description. (Show Details)Dec 19 2022, 4:55 PM
JArguello-WMF set the point value for this task to 9.
JAllemandou renamed this task from [Migration] Learning to [Migration] Pageview - Learning.Dec 20 2022, 11:00 AM
JAllemandou updated the task description. (Show Details)
JAllemandou updated the task description. (Show Details)
• EChetty assigned this task to JAllemandou.Jan 16 2023, 2:29 PM
• EChetty moved this task from To be prioritised to Sprint 07 on the Data Pipelines board.Jan 16 2023, 2:42 PM
• EChetty edited projects, added Data Pipelines (Sprint 07); removed Data Pipelines.
• EChetty moved this task from Ready to Next Up on the Data Pipelines (Sprint 07) board.Jan 16 2023, 4:13 PM
• EChetty moved this task from Next Up to In Progress on the Data Pipelines (Sprint 07) board.Jan 18 2023, 5:12 PM
gerritbot added a comment.Jan 27 2023, 5:45 PM

Change 884359 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery@master] Add hql/webrequest/actor folder and scripts

https://gerrit.wikimedia.org/r/884359

gerritbot added a project: Patch-For-Review.Jan 27 2023, 5:45 PM
gerritbot added a comment.Jan 31 2023, 1:58 PM

joal opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/223

Add analytics webrequest actor dags

JAllemandou moved this task from In Progress to In Review on the Data Pipelines (Sprint 07) board.Jan 31 2023, 1:58 PM
• EChetty moved this task from Sprint 07 to Sprint 08 on the Data Pipelines board.Feb 7 2023, 12:24 PM
• EChetty edited projects, added Data Pipelines (Sprint 08); removed Data Pipelines (Sprint 07).
• EChetty moved this task from Ready to In Review on the Data Pipelines (Sprint 08) board.
JAllemandou moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 08) board.Feb 8 2023, 2:09 PM
gerritbot added a comment.Feb 8 2023, 2:10 PM

Change 887786 had a related patch set uploaded (by Joal; author: Joal):

[operations/puppet@production] Update analytics data purge for webrequest_actor

https://gerrit.wikimedia.org/r/887786

gerritbot added a comment.Feb 8 2023, 6:09 PM

Change 884359 merged by Joal:

[analytics/refinery@master] Add webrequest and pageview actor scripts

https://gerrit.wikimedia.org/r/884359

gerritbot added a comment.Feb 9 2023, 10:20 AM

joal merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/223

Add analytics webrequest and pageview actor dags

JArguello-WMF moved this task from Ready to Deploy to Done on the Data Pipelines (Sprint 08) board.Feb 9 2023, 5:05 PM
gerritbot added a comment.Feb 10 2023, 2:14 PM

Change 887786 merged by Nicolas Fraison:

[operations/puppet@production] Update analytics data purge for webrequest_actor

https://gerrit.wikimedia.org/r/887786

Maintenance_bot removed a project: Patch-For-Review.Feb 10 2023, 2:30 PM
gerritbot added a comment.Feb 10 2023, 3:05 PM

Change 888228 had a related patch set uploaded (by Joal; author: Joal):

[operations/puppet@production] Remove previously absent timers from analytics data_purge

https://gerrit.wikimedia.org/r/888228

gerritbot added a project: Patch-For-Review.Feb 10 2023, 3:05 PM
gerritbot added a comment.Feb 10 2023, 3:16 PM

Change 888228 merged by Nicolas Fraison:

[operations/puppet@production] Remove previously absent timers from analytics data_purge

https://gerrit.wikimedia.org/r/888228

Maintenance_bot removed a project: Patch-For-Review.Feb 10 2023, 3:30 PM
JArguello-WMF closed this task as Resolved.Feb 17 2023, 7:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits