Page MenuHomePhabricator

4 more Kafka brokers, 2 in eqiad and 2 codfw
Closed, DeclinedPublic

Description

In T123954, we have been investigating options for doing cross-DC mirroring of messages in Kafka. On January 21, @Joe, @paravoid, @elukey, @mobrovac and I met and decided that the most ideal solution would be to do what LinkedIn describes here: https://engineering.linkedin.com/kafka/running-kafka-scale. That is, our main Kafka setups in each DC would consist of 2 separate Kafka clusters. The first 'local' cluster is produced to by producers local to a DC. The second 'aggregate' cluster would mirror all data from from clusters in external DCs. Internal consumers could consume from this aggregate cluster in order to receive all messages from all producers in all DCs.

We talked about several ways to scale this infrastructure into the future. We don't technically need this local+aggregate architecture now, but having it allows us to design topics and applications in a more future proof way.

So! I'm requesting 4 more Kafka broker nodes to be ordered, 2 in eqiad and 2 in codfw. They can have the same specs as the nodes chosen in T114191.

Thanks!
-Ao

Event Timeline

Ottomata created this task.Jan 22 2016, 8:13 PM
Ottomata claimed this task.
Ottomata raised the priority of this task from to Needs Triage.
Ottomata updated the task description. (Show Details)
Ottomata edited projects, added hardware-requests; removed Analytics-Cluster, Operations.
Ottomata set Security to None.
Ottomata added subscribers: Joe, fgiunchedi, Krinkle and 11 others.
Restricted Application added a project: Operations. · View Herald TranscriptJan 22 2016, 8:13 PM
RobH claimed this task.Feb 5 2016, 10:45 PM
RobH moved this task from Backlog to In Discussion / Review on the hardware-requests board.
GWicke added a comment.EditedFeb 5 2016, 10:51 PM

I have some questions on this decision:

  1. Which advantages do you see in having separate clusters at this point, considering the relatively limited message volumes expected for EventBus?
  2. Do you intend to merge messages from different clusters into a single Kafka topic, or will all logical topics have several Kafka topics, prefixed / post-fixed with DC name, as discussed in T123954?
RobH reassigned this task from RobH to Ottomata.Feb 6 2016, 12:22 AM
RobH added a subscriber: RobH.

I don't necessarily want to hold up the procurement, but saw the meeting & decision mentioned here for the first time. I think it's worth being clear on the rationales for future reference, which is why I'm asking for a clarification.

  1. Which advantages do you see in having separate clusters at this point, considering the relatively limited message volumes expected for EventBus?

While performance-wise adding the EventBus streams to the existing Analytics cluster wouldn't be a problem, there are other considerations, such the sensitivity of the data and the fact that at some point we will start exposing some channels externally. Also, it is my understanding that Analytics folks do not want to mix apples and oranges at a stage where the oranges aren't fully ripe :)

  1. Do you intend to merge messages from different clusters into a single Kafka topic, or will all logical topics have several Kafka topics, prefixed / post-fixed with DC name, as discussed in T123954?

Each topic will have two logical topics - one for messages produced in the local DC, and another for messages received from all of the other DCs combined.

GWicke added a comment.EditedFeb 6 2016, 9:20 PM

While performance-wise adding the EventBus streams to the existing Analytics cluster wouldn't be a problem

We ruled this possibility out rather early, as the reliability requirements and volumes involved are very different. If I understand @Ottomata right, he is now asking for a second EventBus-only cluster per DC.

Each topic will have two logical topics - one for messages produced in the local DC, and another for messages received from all of the other DCs combined.

This means that we'd have to filter the merged event stream by origin DC when switching the primary event processing on DC fail-over. This seems less efficient & convenient than the original idea of having DC-prefixed / post-fixed topics.

It would also be harder to present a consistent merge after a system wobble, as the merge order per DC would be determined once, at replication time. An (eventually) consistent event order is desirable for a lot of use cases, including analytics.

Ah!

No, this isn't really about eventbus. It is, sorta but it isn't. It is more about planning future cross DC use of the main Kafka clusters.

Do you intend to merge messages from different clusters into a single Kafka topic, or will all logical topics have several Kafka topics, prefixed / post-fixed with DC name, as discussed in T123954?

Messages from different clusters will all be in the same topic. E.g. mediawiki.page_move will be the same in all clusters. This simplifies details producers have to know about what and how to produce and consume.

This ticket asks for secondary 'aggregate' clusters, so that we can have all messages mirrored cross DC in the same topic names. Producers in each DC will produce to their 'local' main Kafka cluster. MirrorMaker in each DC will mirror all messages from all main Kafka clusters into their DC's aggregate Kafka cluster. If a consumer in a given DC needs to consume messages from all DCs, they will consume from their DC's aggregate Kafka cluster.

This is the recommended strategy at LinkedIn, and simplifies topic creation in the future. We would choose not to go this way (and use DC based naming for topics) if Ops didn't want to allocate the hardware for this now. In the meeting we had, we all decided that this was a better way to set ourselves up to be scalable in the future.

will all logical topics have several Kafka topics, prefixed / post-fixed with DC name, as discussed in T123954?

A simpler answer: No. Using aggregate Kafka clusters is what allows us to not have to name topics after DCs.

RobH added a comment.Apr 19 2016, 7:14 PM

irc update: In triaging the #hw-requests, I've checked with @Ottomata. This needs to have further investigation done, so I'm keeping it assigned to him, and not working on this task until he assigns it to me with updates for potential system allocations/ordering.

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 7:14 PM
Krinkle removed a subscriber: Krinkle.Apr 21 2016, 8:46 PM
Ottomata closed this task as Declined.Jul 22 2016, 1:03 PM

No need to do this for now, as long as we have decided to go with datacenter prefixed topic names. Declining.