Page MenuHomePhabricator

EventStreams (via KafkaSSE) does not consume from newly added partitions in topic
Open, LowPublic8 Estimated Story Points

Description

If new partitions are added to a topic, connected KafkaSSE connections will not see that new partition. Connected clients will miss messages.

This is not a common problem, as partitions are rarely added to topics, but still, KafkaSSE should handle this. This is a consequence of the fact that KafkaSSE uses a 'low level' librdkafka consumer via the .assign method, rather than the 'high level' one via the .subscribe method. This was done to avoid saving state on the Kafka brokers for these types of connections.

To fix, KafkaSSE will likely need to periodically issue a getMetadata call, and refresh the assignments (and stored last-event-id) for a connected clients if the partition count changes.

Event Timeline

Ottomata set the point value for this task to 8.
RBrounley_WMF removed Protsack.stephan as the assignee of this task.
RBrounley_WMF added a subscriber: Protsack.stephan.
RBrounley_WMF subscribed.

Hah meant to add @Protsack.stephan as a subscriber not owner

Ottomata added a subscriber: JArguello-WMF.

@JArguello-WMF this is indeed a software bug that will affect users if/when partitions in a Kafka topic are modified. Reopening, I hope that is okay?

@Ottomata More than ok! Thank you for reopening the task.