HomePhabricator

Handle the SSE stream backpressure and promisify it

Authored by mobrovac on Jul 23 2018, 8:20 PM.

Description

Handle the SSE stream backpressure and promisify it

Summary:
Normally, the service is able to read Kafka messages faster than it is
able to deliver them to the client, which can cause memory to build up,
especially in cases where there are a lot of messages to be processed
and the client is notoriously slow. Therefore, build the backpressure
into sending the events. Concretely, do not attempt to read the next
message from Kafka before the current event has been flushed down to the
kernel.

While we are at it, also promisify KafkaSSE properly, so that now most
methods return Promises.

Bug: T199813

Reviewers: Ottomata, elukey, Pchelolo, mobrovac

Reviewed By: mobrovac

Subscribers: jenkins

Differential Revision: https://phabricator.wikimedia.org/D1085

Details

Committed
mobrovacJul 24 2018, 5:59 PM
Reviewer
mobrovac
Differential Revision
D1085: Handle the SSE stream backpressure and promisify it
Parents
rWKSE0e8683d65eea: Update node-rdkafka to v2.3.4
Branches
Unknown
Tags
Unknown
ChangeId
None
Build Status
Buildable 3073
Build 5140: ci-jessieJenkins