Page MenuHomePhabricator

Support per-topic configuration in EventBus service
Closed, DeclinedPublic

Description

If we want to migrate JobQueue to the Event-Platform we'd post all the jobs into Kafka as events. Job parameters could be quite large, so storing the job-related events is not only unnecessary, but really wasteful. Kafka support setting per-topic log.retention.hours, so for job-related topic this should be decreased to 1 day or something. Since job-related topics would be auto created we need to somehow specify that int he topics config.

The job-related topic names should be wild carded so that the retention applies to all the job-related topics. See T157091 for more details.

Event Timeline

Pchelolo created this task.Feb 3 2017, 12:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2017, 12:26 AM
Pchelolo renamed this task from Support per-topic configuration inn EventBus service to Support per-topic configuration in EventBus service.Feb 3 2017, 12:32 AM
GWicke triaged this task as Low priority.Feb 3 2017, 8:34 PM
GWicke added a subscriber: GWicke.

Lets perhaps put this on the backburner until we know we are actually going to be pressed for space?

Nuria moved this task from Incoming to Dashiki on the Analytics board.Feb 9 2017, 4:48 PM
Milimetric raised the priority of this task from Low to Needs Triage.Apr 2 2018, 3:56 PM
Milimetric moved this task from Wikistats Production to Modern Event Platform on the Analytics board.
Milimetric triaged this task as Medium priority.May 3 2018, 5:22 PM
Ottomata closed this task as Declined.Apr 24 2019, 2:57 PM

I don't think this is really needed. If it is, it will be part of T205319: Modern Event Platform: Stream Configuration