Page MenuHomePhabricator

Support per-topic configuration in EventBus service
Closed, DeclinedPublic

Description

If we want to migrate JobQueue to the Event-Platform we'd post all the jobs into Kafka as events. Job parameters could be quite large, so storing the job-related events is not only unnecessary, but really wasteful. Kafka support setting per-topic log.retention.hours, so for job-related topic this should be decreased to 1 day or something. Since job-related topics would be auto created we need to somehow specify that int he topics config.

The job-related topic names should be wild carded so that the retention applies to all the job-related topics. See T157091 for more details.

Event Timeline

Pchelolo renamed this task from Support per-topic configuration inn EventBus service to Support per-topic configuration in EventBus service.Feb 3 2017, 12:32 AM
GWicke subscribed.

Lets perhaps put this on the backburner until we know we are actually going to be pressed for space?

Milimetric raised the priority of this task from Low to Needs Triage.Apr 2 2018, 3:56 PM
Milimetric moved this task from Wikistats to Event Platform on the Analytics board.
Milimetric triaged this task as Medium priority.May 3 2018, 5:22 PM

I don't think this is really needed. If it is, it will be part of T205319: Modern Event Platform: Stream Configuration