Handle the SSE stream backpressure and promisify it
ClosedPublic

Authored by mobrovac on Jul 23 2018, 8:24 PM.

Details

Reviewers
Ottomata
elukey
Pchelolo
mobrovac
Commits
rWKSEbb7e986f39b2: Handle the SSE stream backpressure and promisify it
Patch without arc
git checkout -b D1085 && curl -L https://phabricator.wikimedia.org/D1085?download=true | git apply
Summary

Normally, the service is able to read Kafka messages faster than it is
able to deliver them to the client, which can cause memory to build up,
especially in cases where there are a lot of messages to be processed
and the client is notoriously slow. Therefore, build the backpressure
into sending the events. Concretely, do not attempt to read the next
message from Kafka before the current event has been flushed down to the
kernel.

While we are at it, also promisify KafkaSSE properly, so that now most
methods return Promises.

Bug: T199813

Diff Detail

Repository
rWKSE KafkaSSE
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
mobrovac created this revision.Jul 23 2018, 8:24 PM

Build was aborted

Test Name
npm testsBuild Details

npm tests log

Got exit code 143 from npm run test

Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1128/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1128/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 23 2018, 8:54 PM
Harbormaster failed remote builds in B3067: Diff 2842!
mobrovac updated this revision to Diff 2843.Jul 23 2018, 10:45 PM

remove superfluous return

Build was aborted

Test Name
npm testsBuild Details

npm tests log

Got exit code 143 from npm run test

Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1129/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1129/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 23 2018, 11:15 PM
Harbormaster failed remote builds in B3068: Diff 2843!
mobrovac updated this revision to Diff 2844.Jul 24 2018, 9:32 AM

Send the whole event at once instead of line by line given that the max len never exceeds 32kB

Build was aborted

Test Name

npm tests log

Got exit code 143 from npm run test

Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1130/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1130/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 24 2018, 10:02 AM
Harbormaster failed remote builds in B3069: Diff 2844!
mobrovac updated this revision to Diff 2847.Jul 24 2018, 12:04 PM

Brought back the line-by-line send as SSE uses a line-delimited protocol. Also fixed the assignment mechanism to yield properly.

mobrovac updated this revision to Diff 2849.Jul 24 2018, 12:19 PM

Ensure no writes happen after the SSE stream has been closed

Build was aborted

Test Name
npm testsBuild Details

npm tests log

Got exit code 143 from npm run test

Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1131/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1131/console

Build was aborted

Test Name
npm testsBuild Details

npm tests log

Got exit code 143 from npm run test

Link to build: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1132/
See console output for more information: https://integration.wikimedia.org/ci/job/phabricator-jessie-diffs/1132/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 24 2018, 12:49 PM
Harbormaster failed remote builds in B3071: Diff 2849!
mobrovac updated this revision to Diff 2850.Jul 24 2018, 5:45 PM

Fix the tests

mobrovac requested review of this revision.Jul 24 2018, 5:47 PM
mobrovac accepted this revision.Jul 24 2018, 5:59 PM
This revision is now accepted and ready to land.Jul 24 2018, 5:59 PM
This revision was automatically updated to reflect the committed changes.