Page MenuHomePhabricator
Create Task
Maniphest T111409

Spike replacing Camus with Gobblin {hawk} [13 pts]
Closed, ResolvedPublic
Actions

Description

From Kafka upgrade post-mortem:
https://etherpad.wikimedia.org/p/kafka_upgrade_2015-08_postmortem

Negative results from spike would inform whether we need to pursue:

  1. Make oozie launch load job based on Camus offsets instead of hourly directories in HDFS
  2. Better Camus Monitoring

Related Objects
Search...

Event Timeline

• ggellerman created this task.Sep 3 2015, 5:21 PM
• ggellerman raised the priority of this task from to Needs Triage.
• ggellerman updated the task description. (Show Details)
• ggellerman added projects: Analytics-Clusters, Analytics-Backlog.
• ggellerman added subscribers: • madhuvishy, mforns, Milimetric and 4 others.
Milimetric triaged this task as High priority.Sep 10 2015, 5:12 PM
Milimetric set Security to None.
Milimetric renamed this task from Spike replacing Camus with Gobblin to Spike replacing Camus with Gobblin {hawk}.Sep 10 2015, 5:14 PM
Milimetric moved this task from Incoming to Prioritized on the Analytics-Backlog board.
• kevinator added a subscriber: • kevinator.Sep 14 2015, 4:54 PM

documentation on Goblin:
https://github.com/linkedin/gobblin/wiki/Kafka-HDFS-Ingestion

Milimetric renamed this task from Spike replacing Camus with Gobblin {hawk} to Spike replacing Camus with Gobblin {hawk} [13 pts].Sep 14 2015, 4:57 PM
Milimetric moved this task from Prioritized to Tasked on the Analytics-Backlog board.
• madhuvishy claimed this task.Sep 16 2015, 12:56 AM
• madhuvishy edited projects, added Analytics-Kanban; removed Analytics-Backlog.
• madhuvishy moved this task from Next Up to In Progress on the Analytics-Kanban board.
• kevinator added a parent task: T109192: Implement better Webrequest load monitoring {hawk}.Sep 21 2015, 4:22 PM
• madhuvishy added a comment.Sep 21 2015, 9:05 PM

https://etherpad.wikimedia.org/p/gobblin-sprint

• madhuvishy moved this task from In Progress to Done on the Analytics-Kanban board.Sep 21 2015, 9:14 PM
• kevinator closed this task as Resolved.Sep 22 2015, 4:08 PM

We're going to keep camus for now, but if we need more features, we'll switch to Goblin in the future.

elukey mentioned this in T238400: Evaluate possible replacements for Camus: Gobblin, Marmaray, Kafka Connect HDFS, etc..Nov 15 2019, 10:04 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL