In JobQueue we have one queue per job type, so we need a separate Kafka topic per job type.
In Event-Platform all of these topics will have the same schema (some kind of a job schema) and kafka supports automatic topic creation. To avoid duplicating the config we need to support wildcards (regexes) in the EventBus schema config to do something like job_queue.*: job_schema
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Pchelolo | T157088 [EPIC] Develop a JobQueue backend based on EventBus | |||
| Resolved | Ottomata | T157091 Make EventBus service support wildcards in schema definitions |
@Ottomata would you have any spare cycles to work on this or to guide me a bit through the event logging code to see how to do that? I'm slowly progressing on the general task and this seems to be a logical next preparation step..
Sure! If you want to take a look, this is handled by eventlogging/topic.py in the function schema_name_for_topic.
You'll have to add some regex matching in there I think. I'm pretty sure that should be it.
All right @Ottomata, I'll try to remember python which I didn't use in like 5 years, let's see where it gets us :)
Change 348805 had a related patch set uploaded (by Ppchelko):
[eventlogging@master] WIP: Support regexed topic definitions
Change 348861 had a related patch set uploaded (by Ppchelko):
[mediawiki/event-schemas@master] Topic-Config: Use regex topic definitions for rerty topics.
Change 348805 merged by Ottomata:
[eventlogging@master] Support regexed topic definitions
Change 348861 merged by jenkins-bot:
[mediawiki/event-schemas@master] Topic-Config: Use regex topic definitions for rerty topics.