Page MenuHomePhabricator

Improve tests for KafkaSSE
Closed, ResolvedPublic13 Story Points

Description

KafkaSSE tests work locally, but don't run well in Jenkins/Travis because a Kafka broker is spawned.

Goal is to test integration with Kafka client node-rdkafka

We want to identify changes to Kafka SSE that break integration with node-rdkafka

[nuria] jenkins spwning a broker might not be the best way. Can we provision a docker environment with all that is needed and use it locally w/o involving jenkins?

Event Timeline

Ottomata created this task.Nov 10 2016, 3:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 10 2016, 3:46 PM
Nuria added a comment.Nov 10 2016, 4:50 PM

Several Options:

  • kafka broker tests could be run just locally
  • mocks could be added to test jenkins with kafka

Or:

  • we could beef up jenkins kafka setup
Nuria changed the point value for this task from 5 to 13.Nov 10 2016, 4:51 PM
Nuria edited projects, added Analytics-Kanban; removed Analytics.
mforns claimed this task.Nov 30 2016, 3:53 PM
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.
Nuria updated the task description. (Show Details)Dec 1 2016, 5:28 PM
hashar added a subscriber: hashar.

I am merely brain dumping my understanding for this task:

node-rdkafka are nodejs bindings for librdkafka. It can be compiled either against the system package or a source package. The canonical repository is https://github.com/wikimedia/node-rdkafka

It seems KafkaSSE canonical repository is on Differential https://phabricator.wikimedia.org/diffusion/WKSE/

There is some discussion going on T151469 to eventually get Kafka to spawn in jobs and rely on it to run integration tests.

For KafkaSSE, I guess you would install node-rdkafka from npmjs.org, start Kafka and run tests against it. If one want to test against versions that runs on production that would be a different Job which would have to fetch the version of node-rdkafka currently deployed and compiled against system package. Then Nuria pointed that last part would be better suited for a staging environment.

mforns reassigned this task from mforns to JAllemandou.Dec 19 2016, 4:13 PM

I just landed Joseph's docker KafkaSSE test patch. We could make some improvements to it to make it better for running tests during developments, but it should be good as is for Jenkins...I think.

Nuria moved this task from In Code Review to Done on the Analytics-Kanban board.Jan 19 2017, 5:06 PM
Nuria closed this task as Resolved.Jan 24 2017, 9:59 PM