Page MenuHomePhabricator
Create Task
Maniphest T128611

Modify Camus file commits to use OVERWRITE option when renaming into final destination
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

If Camus's offset files get wonky, it may have to re run a job and reimport data from Kafka. In this case, it may need to overwrite files in the final destination. However, currently if a file in the final destination already exists, and the rename() fails, the whole Camus job will fail, not update history offsets, and get stuck.

We should modify the camus commit to allow it to overwrite files. It looks like switching the [[ https://github.com/wikimedia/analytics-camus/blob/wmf/camus-etl-kafka/src/main/java/com/linkedin/camus/etl/kafka/mapred/EtlMultiOutputCommitter.java#L153 | rename() call in EltMultiOutputCommiter commitFile ]] from [[ https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/fs/FileSystem.html#rename(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path) | FileSystem ]] to [[ http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html#rename(org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Options.Rename...) | FileContext ]] and using the OVERWRITE option should do it.

Details

SubjectRepoBranchLines +/-
Make camus overwrite exisiting filesoperations/puppetproduction+3 -0
Add destination overwriting capacity to camusanalytics/camusmaster+58 -4
Add destination overwriting capacity to camusanalytics/camuswmf+58 -4
Customize query in gerrit

Event Timeline

Ottomata created this task.Mar 2 2016, 5:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 2 2016, 5:11 PM
JAllemandou triaged this task as High priority.Mar 2 2016, 5:55 PM
JAllemandou edited projects, added Analytics-Kanban; removed Analytics.
gerritbot subscribed.Mar 3 2016, 3:46 PM

Change 274736 had a related patch set uploaded (by Joal):
Add destination overwriting capacity to camus

https://gerrit.wikimedia.org/r/274736

gerritbot added a project: Patch-For-Review.Mar 3 2016, 3:46 PM
JAllemandou claimed this task.Mar 3 2016, 4:48 PM
JAllemandou moved this task from Next Up to In Code Review on the Analytics-Kanban board.
Milimetric set the point value for this task to 5.Mar 3 2016, 5:12 PM
gerritbot added a comment.Mar 4 2016, 7:20 PM

Change 274736 merged by Ottomata:
Add destination overwriting capacity to camus

https://gerrit.wikimedia.org/r/274736

gerritbot added a comment.Mar 4 2016, 8:35 PM

Change 275047 had a related patch set uploaded (by Ottomata):
Add destination overwriting capacity to camus

https://gerrit.wikimedia.org/r/275047

gerritbot added a comment.Mar 4 2016, 8:36 PM

Change 275047 merged by Ottomata:
Add destination overwriting capacity to camus

https://gerrit.wikimedia.org/r/275047

JAllemandou moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Mar 7 2016, 10:28 AM
JAllemandou moved this task from Ready to Deploy to In Code Review on the Analytics-Kanban board.Mar 9 2016, 1:12 PM
gerritbot added a comment.Mar 9 2016, 2:24 PM

Change 276165 had a related patch set uploaded (by Joal):
Make camus overwrite exisiting files

https://gerrit.wikimedia.org/r/276165

gerritbot added a comment.Mar 9 2016, 2:39 PM

Change 276165 merged by Ottomata:
Make camus overwrite exisiting files

https://gerrit.wikimedia.org/r/276165

JAllemandou moved this task from In Code Review to Done on the Analytics-Kanban board.Mar 9 2016, 2:40 PM
• Nuria closed this task as Resolved.Apr 5 2016, 8:05 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL