Page MenuHomePhabricator

camus offset fails/continues Load job {hawk} [13 pts]
Closed, ResolvedPublic

Description

Change load job to fail/continue based on camus most recent run offset files.
The job reading and computing offsets from camus history files will touch the _PARTITIONED file to manifest the fact that the partition is (normally) fully imported.
The current oozie load job needs to be changed to make use of this file to start, and use a new file (_CHECKED?) for statistics checking.

Event Timeline

kevinator assigned this task to Ottomata.
kevinator raised the priority of this task from to Normal.
kevinator updated the task description. (Show Details)
kevinator moved this task from Incoming to Tasked on the Analytics-Backlog board.
kevinator removed Ottomata as the assignee of this task.Sep 21 2015, 5:01 PM
kevinator set Security to None.

Hey, I was thinking about this more after talking with Madhu about the Gobblin vs. Camus spike yesterday, and I think I'd like to change this task. Will talk in standup today.

JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.
JAllemandou renamed this task from webrequest_sequence_stats_hourly fails/continues Load job {hawk} [13 pts] to camus offset fails/continues Load job {hawk} [13 pts].Sep 29 2015, 1:56 PM
JAllemandou claimed this task.
JAllemandou updated the task description. (Show Details)

Change 248048 had a related patch set uploaded (by Joal):
Update bin/camus to include CamusPartitionChecker

https://gerrit.wikimedia.org/r/248048

Change 249100 had a related patch set uploaded (by Joal):
Update camus runs

https://gerrit.wikimedia.org/r/249100

Change 248048 merged by Ottomata:
Update bin/camus to include CamusPartitionChecker

https://gerrit.wikimedia.org/r/248048

Change 249100 merged by Ottomata:
Update camus runs

https://gerrit.wikimedia.org/r/249100

kevinator closed this task as Resolved.Oct 29 2015, 8:40 PM