Page MenuHomePhabricator

Puppetize and make useable confluent kafka packages
Closed, ResolvedPublic21 Estimated Story Points

Description

It is going to be very difficult to update our custom Kafka .deb package to Kakfa 0.9

  • Scala 2.9 is no longer supported. (This is what Debian ships)
  • Lots of kafka shell script changes.
  • MirrorMaker no longer supports multiple target clusters. etc.

Instead, we will adapt our puppet and internal usage of Kafka to use the Confluent provided packages.

Event Timeline

Ottomata created this task.Apr 13 2016, 9:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 13 2016, 9:06 PM
Ottomata updated the task description. (Show Details)Apr 13 2016, 9:07 PM

Ok, after some discussion on the ops list, it seems the best option is to use the confluent provided deb package instead of continuing to maintain our own.

Changing ticket accordingly.

Ottomata renamed this task from Debianize Kafka 0.9 to Puppetize and make useable confluent kafka packages.Apr 14 2016, 8:19 PM
Ottomata edited projects, added Analytics-Kanban; removed Analytics.
Ottomata updated the task description. (Show Details)
Ottomata set the point value for this task to 21.

Change 284349 had a related patch set uploaded (by Ottomata):
[WIP] Add new confluent module and puppetization for using confluent Kafka

https://gerrit.wikimedia.org/r/284349

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 8:56 PM

For my own reference, a few defaults have changed in Kafka's server.properties.erb for 0.9. I'm making these changes to the defaults in the confluent puppet module. I need to remember to set them appropriately to what we have in production now when we make the change to use the confluent module. These are:

$auto_create_topics_enable           = true,       # was false in our kafka module
$log_segment_bytes                   = 1073741824, # was 536870912 in our kafka module.

Change 284349 merged by Ottomata:
Add new confluent module and puppetization for using confluent Kafka

https://gerrit.wikimedia.org/r/284349

Ottomata moved this task from In Code Review to Done on the Analytics-Kanban board.May 5 2016, 4:05 PM
Nuria closed this task as Resolved.May 11 2016, 3:45 PM