Page MenuHomePhabricator

EventGate should be able to configure hasty and guaranteed kafka producers individually
Closed, ResolvedPublic3 Story Points

Description

EventGate app configs currently use a single kafka.conf for both producer types. Since the hasty producer does not wait on a delivery report before finishing the HTTP response, it can use a much higher value of queue.buffering.max.ms. However a longer queue.buffering.max.ms will cause a the guaranteed producer to likely block for about that amount of time. We should set the hasty producer with a high value, and the guaranteed producer with a lower value.

Event Timeline

Ottomata created this task.Mar 22 2019, 8:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 22 2019, 8:25 PM
fdans moved this task from Incoming to Modern Event Platform on the Analytics board.
fdans triaged this task as High priority.
Ottomata moved this task from Backlog to Done on the EventBus board.Mar 26 2019, 1:53 PM

Change 499204 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - use different batch settings for kafka producers

https://gerrit.wikimedia.org/r/499204

Change 499204 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - use different batch settings for kafka producers

https://gerrit.wikimedia.org/r/499204

Nuria moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Apr 9 2019, 4:03 PM
Ottomata set the point value for this task to 3.Apr 10 2019, 4:01 PM
Ottomata closed this task as Resolved.Apr 24 2019, 2:44 PM