Page MenuHomePhabricator

Refactor kafka_config.rb and and kafka_cluster_name.rb in puppet to avoid explicit hiera calls
Closed, DeclinedPublic

Description

kafka_config.rb and and kafka_cluster_name.rb are two useful functions in operations/puppet that transparently retrieve from hiera all the necessary configuration required by a Kafka consumer/producer. This code is now colliding a bit with the new profile/role convention, because in theory no explicit hiera call is allowed anymore ouside a profile's parameters.

Ideally these functions could become stateless simply adding new parameters instead of retrieving those values via hiera. This would require more hiera configurations but it would still allow code reuse among Kafka consumers/producers configurations.

Event Timeline

ema triaged this task as Medium priority.Oct 11 2017, 1:25 PM
ema added a project: Traffic.
ema subscribed.

This would really only require passing kafka_clusters as well as kafka_cluster_name to the kafka_config() function, instead of looking it up from within the function? I think this sounds good, and would make testing / reuse of the function a little easier too.

Milimetric moved this task from Dashiki to Incoming on the Analytics board.

Hm, I just thought about this a little bit, and I'm not so sure we should do it. The hiera info that this function is looking up is always at the global common.yaml level. There is never a case in given environment (e.g. labs vs prod) where the value of kafka_clusters is different per role or node or whatever.

I'd be fine with these two ruby functions to take an extra optional parameter with kafka_cluster config, as this would make it easier to test. However if not provided, these functions should lookup hiera('kafka_clusters') by default. I don't see any benefit to parameterizing something that will never vary.

mforns lowered the priority of this task from Medium to Low.Oct 15 2018, 4:39 PM

Old task, I think we aren't likely to do this. Declining, feel free to reopen if wrong.