Some streams have decided to adopt a double compute, active/active approach.
In such scenario the the kafka topics defined are logical replicas of each others and are not complementing themselves like most of our streams that are running single compute.
The EventStreams HTTP service has been built with this assumption and thus is not yet suited to serve double compute streams.
Proposal
Assuming a double compute stream defined as follow
name: my-stream.v1 topics: - eqiad.my-stream.v1 - codfw.my-stream.v1
The EventStreams HTTP service can allow overriding the stream definitions on a per DC basis (values-codfw.yaml vs values-eqiad.yaml)
values-eqiad.yaml :
stream_config_defaults: my-stream.v1: topics: - eqiad.my-stream.v1 topics_allowed: - codfw.my-stream.v1 - eqiad.my-stream.v1
values-codfw.yaml:
stream_config_defaults: my-stream.v1: topics: - codfw.my-stream.v1 topics_allowed: - codfw.my-stream.v1 - eqiad.my-stream.v1
Operations
A client accessing the service running eqiad will only consume eqiad.my-stream.v1, the client will receive a Last-Event-ID of the form:
[ {"topic": "eqiad.my-stream.v1", "partition": 0, "timestamp": 1575906290000}, {"topic": "eqiad.my-stream.v1", "partition": 1, "timestamp": 1575906290000} ]
(we use useTimestampForId)
To resume operations after the connection ends.
Resuming from the same DC is not an issue since the topic should match.
Resuming from another DC the service should continue to serve data from that topic eventhough this is not the default one but one allowed in the topics_allowed array.
For instance:
the client sends a request to codfw with:
[ {"topic": "eqiad.my-stream.v1", "partition": 0, "timestamp": 1575906290000}, {"topic": "eqiad.my-stream.v1", "partition": 1, "timestamp": 1575906290000} ]
The service should understand that given the configuration:
stream_config_defaults: my-stream.v1: topics: - codfw.my-stream.v1 topics_allowed: - eqiad.my-stream.v1 - codfw.my-stream.v1
eqiad.my-stream.v1 is allowed and can continue consuming from this topic even though this is not the default topic a client should consume from when hitting codfw.
AC:
- a double compute stream can be configured in the eventstreams HTTP endpoint
- a consumer should not fail if it tries to resume consumption on a topic that's a replica of the topic populated in that DC