Page MenuHomePhabricator

EventGate should be able to configure hasty and guaranteed kafka producers individually
Closed, ResolvedPublic3 Story Points

Description

EventGate app configs currently use a single kafka.conf for both producer types. Since the hasty producer does not wait on a delivery report before finishing the HTTP response, it can use a much higher value of queue.buffering.max.ms. However a longer queue.buffering.max.ms will cause a the guaranteed producer to likely block for about that amount of time. We should set the hasty producer with a high value, and the guaranteed producer with a lower value.

Event Timeline

Ottomata created this task.Mar 22 2019, 8:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 22 2019, 8:25 PM
fdans triaged this task as High priority.Mar 25 2019, 3:48 PM
fdans added a project: Analytics-Kanban.
fdans moved this task from Incoming to Modern Event Platform on the Analytics board.
Ottomata moved this task from Backlog to Done on the EventBus board.Mar 26 2019, 1:53 PM

Change 499204 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - use different batch settings for kafka producers

https://gerrit.wikimedia.org/r/499204

Change 499204 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - use different batch settings for kafka producers

https://gerrit.wikimedia.org/r/499204

Nuria moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Apr 9 2019, 4:03 PM
Ottomata set the point value for this task to 3.Apr 10 2019, 4:01 PM
Ottomata closed this task as Resolved.Apr 24 2019, 2:44 PM