Page MenuHomePhabricator

EventGate should be able to configure hasty and guaranteed kafka producers individually
Closed, ResolvedPublic3 Estimated Story Points

Description

EventGate app configs currently use a single kafka.conf for both producer types. Since the hasty producer does not wait on a delivery report before finishing the HTTP response, it can use a much higher value of queue.buffering.max.ms. However a longer queue.buffering.max.ms will cause a the guaranteed producer to likely block for about that amount of time. We should set the hasty producer with a high value, and the guaranteed producer with a lower value.

Event Timeline

fdans added a project: Analytics-Kanban.
fdans moved this task from Incoming to Event Platform on the Analytics board.

Change 499204 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - use different batch settings for kafka producers

https://gerrit.wikimedia.org/r/499204

Change 499204 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - use different batch settings for kafka producers

https://gerrit.wikimedia.org/r/499204

Ottomata set the point value for this task to 3.Apr 10 2019, 4:01 PM