Page MenuHomePhabricator

Make EventBus service support wildcards in schema definitions
Closed, ResolvedPublic

Description

In JobQueue we have one queue per job type, so we need a separate Kafka topic per job type.

In EventBus all of these topics will have the same schema (some kind of a job schema) and kafka supports automatic topic creation. To avoid duplicating the config we need to support wildcards (regexes) in the EventBus schema config to do something like job_queue.*: job_schema

Event Timeline

Pchelolo created this task.Feb 3 2017, 12:21 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2017, 12:21 AM
Nuria moved this task from Incoming to Dashiki on the Analytics board.Feb 9 2017, 4:48 PM

@Ottomata would you have any spare cycles to work on this or to guide me a bit through the event logging code to see how to do that? I'm slowly progressing on the general task and this seems to be a logical next preparation step..

Sure! If you want to take a look, this is handled by eventlogging/topic.py in the function schema_name_for_topic.

You'll have to add some regex matching in there I think. I'm pretty sure that should be it.

All right @Ottomata, I'll try to remember python which I didn't use in like 5 years, let's see where it gets us :)

Change 348805 had a related patch set uploaded (by Ppchelko):
[eventlogging@master] WIP: Support regexed topic definitions

https://gerrit.wikimedia.org/r/348805

Change 348861 had a related patch set uploaded (by Ppchelko):
[mediawiki/event-schemas@master] Topic-Config: Use regex topic definitions for rerty topics.

https://gerrit.wikimedia.org/r/348861

Change 348805 merged by Ottomata:
[eventlogging@master] Support regexed topic definitions

https://gerrit.wikimedia.org/r/348805

The patch was merged, to be deployed on Monday

Pchelolo closed this task as Resolved.Apr 25 2017, 5:24 PM

Deployed to production, resolving.

Change 348861 merged by jenkins-bot:
[mediawiki/event-schemas@master] Topic-Config: Use regex topic definitions for rerty topics.

https://gerrit.wikimedia.org/r/348861