For T160748, I've been trying to think of a nice way to keep topic, schema, and EventStream routes DRY and sane. Currently:
- schemas and topic -> schema mappings are in mediawiki/event-schemas repository
- EventStreams route -> topic mapping in in Puppet hiera
- EventStreams spec.yaml.j2 and config.yaml.j2 templates are in scap,
Scap renders the the spec.yaml and config.yaml files for the EventStreams service, which originally source their data from puppet hiera. I'd like to add schema URIs to the rendered spec.yaml, so that swagger-ui docs can display the schema for each stream route.
I just talked with @Pchelolo a bit, and we think we'll have some more needs to have topic configuration in puppet soon enough, due to the upcoming JobQueue rewrite.
We think we should come up with a good and extensible topic config data structure that we can store in hiera, that would specify things like:
- the schemas allowed to be produced to the topic (if a schema is required).
- other Kafka topic details: number of partitions, retention, etc. (This could be hard to ensure with puppet)
- a description of what the topic is for(?)
I could then use this config to render EventStreams spec.yaml and config.yaml. I'd only have to provide the route name and the topics that should be included in that route. The rest would be looked up by getting the topic config for each topic.
@mobrovac, whatcha think?