Page MenuHomePhabricator

Improve tests for KafkaSSE
Closed, ResolvedPublic13 Estimated Story Points


KafkaSSE tests work locally, but don't run well in Jenkins/Travis because a Kafka broker is spawned.

Goal is to test integration with Kafka client node-rdkafka

We want to identify changes to Kafka SSE that break integration with node-rdkafka

[nuria] jenkins spwning a broker might not be the best way. Can we provision a docker environment with all that is needed and use it locally w/o involving jenkins?

Event Timeline

Several Options:

  • kafka broker tests could be run just locally
  • mocks could be added to test jenkins with kafka


  • we could beef up jenkins kafka setup
Nuria changed the point value for this task from 5 to 13.Nov 10 2016, 4:51 PM
Nuria edited projects, added Analytics-Kanban; removed Analytics.
hashar subscribed.

I am merely brain dumping my understanding for this task:

node-rdkafka are nodejs bindings for librdkafka. It can be compiled either against the system package or a source package. The canonical repository is

It seems KafkaSSE canonical repository is on Differential

There is some discussion going on T151469 to eventually get Kafka to spawn in jobs and rely on it to run integration tests.

For KafkaSSE, I guess you would install node-rdkafka from, start Kafka and run tests against it. If one want to test against versions that runs on production that would be a different Job which would have to fetch the version of node-rdkafka currently deployed and compiled against system package. Then Nuria pointed that last part would be better suited for a staging environment.

I just landed Joseph's docker KafkaSSE test patch. We could make some improvements to it to make it better for running tests during developments, but it should be good as is for Jenkins...I think.