Change Details

EventStreams is backed by the analytics Kafka cluster, which has its topics mirrored from main Kafka. EventStreams could run on main Kafka, which would give it better multi DC support and make it no longer rely on MirrorMaker. We need to move EventStreams' Kafka from analytics to multi DC main. This will mean that offsets will change, which will disrupt connected SSE/EventSource clients. When they reconnect, they will provide invalid offsets to EventStreams. Fortunetly, node-rdkafka will just disregard these invalid offsets and start consuming at the end of the stream. This means that during the switchover, clients may lose a few messages. Since this will effectively connect internet based users to the main Kafka clusters, we should implement some simple connection limits in EventStreams. Anyway, for EventStreams: [x] {T196553} [] Merge [[ https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/439996/ | puppet patch ]], bounce EventStreams.

EventStreams is backed by the analytics Kafka cluster, which has its topics mirrored from main Kafka. EventStreams could run on main Kafka, which would give it better multi DC support and make it no longer rely on MirrorMaker. We need to move EventStreams' Kafka from analytics to multi DC main. This will mean that offsets will change, which will disrupt connected SSE/EventSource clients. When they reconnect, they will provide invalid offsets to EventStreams. Fortunetly, node-rdkafka will just disregard these invalid offsets and start consuming at the end of the stream. This means that during the switchover, clients may lose a few messages. Since this will effectively connect internet based users to the main Kafka clusters, we should implement some simple connection limits in EventStreams. Anyway, for EventStreams: [x] {T196553} [x] Merge [[ https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/439996/ | puppet patch ]], bounce EventStreams.

EventStreams is backed by the analytics Kafka cluster, which has its topics mirrored from main Kafka. EventStreams could run on main Kafka, which would give it better multi DC support and make it no longer rely on MirrorMaker. We need to move EventStreams' Kafka from analytics to multi DC main. This will mean that offsets will change, which will disrupt connected SSE/EventSource clients. When they reconnect, they will provide invalid offsets to EventStreams. Fortunetly, node-rdkafka will just disregard these invalid offsets and start consuming at the end of the stream. This means that during the switchover, clients may lose a few messages. Since this will effectively connect internet based users to the main Kafka clusters, we should implement some simple connection limits in EventStreams. Anyway, for EventStreams: [x] {T196553} [x] Merge [[ https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/439996/ | puppet patch ]], bounce EventStreams.