| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Ottomata | T213602 virtualpageview_hourly lacks data from December 17 on | |||
| Resolved | None | T213716 Alarms for virtualpageview should exist (probably in oozie) for jobs that have been idle too long |
Event Timeline
As FYI I can see the following for Dec 17th in my inbox for analytics-alerts@:
OOZIE - SLA END_MISS (AppName=virtualpageview-hourly-coord, JobID=0001715-180905070129339-oozie-oozi-C@2450) Status: SLA Status - END_MISS Job Status - WAITING Job Details: App Name - virtualpageview-hourly-coord App Type - COORDINATOR_ACTION User - hdfs Job ID - 0001715-180905070129339-oozie-oozi-C@2450 Job URL - http://an-coord1001.eqiad.wmnet:11000/oozie/?job=0001715-180905070129339-oozie-oozi-C@2450 Parent Job ID - 0001715-180905070129339-oozie-oozi-C Parent Job URL - http://an-coord1001.eqiad.wmnet:11000/oozie/?job=0001715-180905070129339-oozie-oozi-C SLA Details: Nominal Time - Mon Dec 17 13:40:00 UTC 2018 Expected End Time - Mon Dec 17 19:40:00 UTC 2018 Expected Duration (in mins) - -1 Actual Duration (in mins) - -1
As far as I can see it we missed to follow up, so probably Oozie should keep sending emails when a job is waiting for too long?
The hue dashboard for workflows displays SLAS (probably not big news) . Looked at docs and i saw we can add an alarm for durantion of job which "seems" to be different than sla for end time. Now, using oozie's sla bindings there is no way to send repeated alarms .
I have a suggestion. We could set the <timeout>XXX</timeout> control in oozie coordinators, replacing XXX with number of seconds before the materialized job times-out. When a job times-out it is marked as failed by oozie, sending us a failure email. I think an error email would make react stronger, as it means data will be missing. Also, the action might just be to re-run the job, which is simple in hue. Finally, this is the approach we have in webrequest-load, which has proven succesfull.