I have some questions on this decision:

1. Which advantages do you see in having separate clusters at this point, considering the relatively limited message volumes expected for EventBus?
2. Do you intend to merge messages from different clusters into a single Kafka topic, or will all logical topics have several Kafka topics, prefixed / post-fixed with DC name, as discussed in T123954?

I don't necessarily want to hold up the procurement, but saw the meeting & decision mentioned here for the first time. I think it's worth being clear on the rationales for future reference, which is why I'm asking for a clarification.

In T124469#2004185, @GWicke wrote:

1. Which advantages do you see in having separate clusters at this point, considering the relatively limited message volumes expected for EventBus?

While performance-wise adding the EventBus streams to the existing Analytics cluster wouldn't be a problem, there are other considerations, such the sensitivity of the data and the fact that at some point we will start exposing some channels externally. Also, it is my understanding that Analytics folks do not want to mix apples and oranges at a stage where the oranges aren't fully ripe :)

2. Do you intend to merge messages from different clusters into a single Kafka topic, or will all logical topics have several Kafka topics, prefixed / post-fixed with DC name, as discussed in T123954?

Each topic will have two logical topics - one for messages produced in the local DC, and another for messages received from all of the other DCs combined.

While performance-wise adding the EventBus streams to the existing Analytics cluster wouldn't be a problem

We ruled this possibility out rather early, as the reliability requirements and volumes involved are very different. If I understand @Ottomata right, he is now asking for a second EventBus-only cluster per DC.

Each topic will have two logical topics - one for messages produced in the local DC, and another for messages received from all of the other DCs combined.

This means that we'd have to filter the merged event stream by origin DC when switching the primary event processing on DC fail-over. This seems less efficient & convenient than the original idea of having DC-prefixed / post-fixed topics.

It would also be harder to present a consistent merge after a system wobble, as the merge order per DC would be determined once, at replication time. An (eventually) consistent event order is desirable for a lot of use cases, including analytics.

Ah!

No, this isn't really about eventbus. It is, sorta but it isn't. It is more about planning future cross DC use of the main Kafka clusters.

Do you intend to merge messages from different clusters into a single Kafka topic, or will all logical topics have several Kafka topics, prefixed / post-fixed with DC name, as discussed in T123954?

Messages from different clusters will all be in the same topic. E.g. mediawiki.page_move will be the same in all clusters. This simplifies details producers have to know about what and how to produce and consume.

This ticket asks for secondary 'aggregate' clusters, so that we can have all messages mirrored cross DC in the same topic names. Producers in each DC will produce to their 'local' main Kafka cluster. MirrorMaker in each DC will mirror all messages from all main Kafka clusters into their DC's aggregate Kafka cluster. If a consumer in a given DC needs to consume messages from all DCs, they will consume from their DC's aggregate Kafka cluster.

This is the recommended strategy at LinkedIn, and simplifies topic creation in the future. We would choose not to go this way (and use DC based naming for topics) if Ops didn't want to allocate the hardware for this now. In the meeting we had, we all decided that this was a better way to set ourselves up to be scalable in the future.

will all logical topics have several Kafka topics, prefixed / post-fixed with DC name, as discussed in T123954?

A simpler answer: No. Using aggregate Kafka clusters is what allows us to not have to name topics after DCs.

irc update: In triaging the #hw-requests, I've checked with @Ottomata. This needs to have further investigation done, so I'm keeping it assigned to him, and not working on this task until he assigns it to me with updates for potential system allocations/ordering.