Page MenuHomePhabricator
Create Task
Maniphest T222603

Fix oozie banner_impression monthly job
Closed, ResolvedPublic1 Estimated Story Points
Actions

Description

The oozie banner-impression monthly coordinator has been failing for the past 10 instances.
The failures seem due to timeouts, due to the lack of of the 90 days postponing _SUCCESS file (the jobs needs 1 month of success flags for data to be present, and another one 90 days after the current month, to prevent sanitizing data too fast). The problem is that daily folders are deleted after 90 days, deleting in the meantime the _SUCCESS flags, therefore preventing the jobs to run. We need to find a new strategy here. The data being dropped is not an issue for druid as the job is re-indexing using already indexed data (segments).

Details

SubjectRepoBranchLines +/-
Fix XML namespace for SLA in banner_monthly jobanalytics/refinerymaster+2 -0
Fix oozie banner monthly jobanalytics/refinerymaster+25 -2
Customize query in gerrit

Related Objects

Event Timeline

JAllemandou created this task.May 6 2019, 11:22 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 6 2019, 11:22 AM
JAllemandou added a comment.May 6 2019, 11:23 AM

A manual fix has been applied to 2018 jobs.

JAllemandou updated the task description. (Show Details)May 6 2019, 11:23 AM
elukey subscribed.May 6 2019, 2:35 PM

Please also restart the job as user analytics, not hdfs (related patch is https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/508283/, if not deployed when you restart please use the -D user=analytics override and start the job via sudo -u analytics oozie etc..)

Milimetric triaged this task as High priority.May 6 2019, 3:36 PM
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.
Milimetric added subscribers: • DStrine, AndyRussG, Milimetric.May 6 2019, 3:52 PM

ping @DStrine do you have a timeline to move data collection for this schema to the EventLogging pipeline? We are fixing this job this time, but it's a duplicate of our standard pipeline that we don't want to maintain going forward. cc @AndyRussG

JAllemandou claimed this task.May 6 2019, 4:14 PM
JAllemandou set the point value for this task to 1.
JAllemandou moved this task from Next Up to In Code Review on the Analytics-Kanban board.
gerritbot added a comment.May 6 2019, 4:15 PM

Change 508358 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Fix oozie banner monthly job

https://gerrit.wikimedia.org/r/508358

gerritbot added a project: Patch-For-Review.May 6 2019, 4:15 PM
• DStrine added a comment.May 6 2019, 4:28 PM

We will get back to event logging for CentralNotice soon but I don't have a timeline. We might have more information in a month or two.

JAllemandou moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.May 7 2019, 5:28 PM
mforns moved this task from Ready to Deploy to Done on the Analytics-Kanban board.May 9 2019, 6:48 PM
gerritbot added a comment.May 13 2019, 9:22 AM

Change 508358 merged by Joal:
[analytics/refinery@master] Fix oozie banner monthly job

https://gerrit.wikimedia.org/r/508358

JAllemandou moved this task from Done to Ready to Deploy on the Analytics-Kanban board.May 13 2019, 9:22 AM
gerritbot added a comment.May 16 2019, 8:08 PM

Change 510773 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Fix XML namespace for SLA in banner_monthly job

https://gerrit.wikimedia.org/r/510773

gerritbot added a comment.May 16 2019, 8:12 PM

Change 510773 merged by Joal:
[analytics/refinery@master] Fix XML namespace for SLA in banner_monthly job

https://gerrit.wikimedia.org/r/510773

JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.May 17 2019, 6:06 PM
Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 3:13 PM
AndyRussG mentioned this in T186048: Adapt ingress of CN data into Druid to EventLogging-based impression recording.May 22 2019, 4:37 PM
• Nuria closed this task as Resolved.Jun 17 2019, 7:58 PM
Ottomata mentioned this in T264358: Investigate oozie banner monthly job timeouts.Nov 30 2020, 7:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL