Page MenuHomePhabricator

Spike: Consider alternatives to MirrorMaker: uReplicator, Confluent Replicator
Closed, ResolvedPublic13 Estimated Story Points

Description

MirrorMaker has always been a little flaky (requiring manual restarts to get proper partition rebalance assignment, etc.), but is being especially so when replicating from the 0.9 main Kafka cluster to the new 1.0 jumbo Kafka cluster. Other's have also experienced this MirrorMaker flakiness. We should try out other options and consider switching, especially since T189464 now blocks the Kafka jumbo migration tasks.

  • Confluent Replicator
    • Closed sourced, but still free
    • Just a Kafka Connect connector
    • Latest Kafka, backed by Confluent

Event Timeline

Ottomata triaged this task as High priority.Mar 19 2018, 2:31 PM
Ottomata created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 19 2018, 2:31 PM

From https://github.com/uber/uReplicator/issues/8#issuecomment-375104514

uReplicator consumer is using 0.10.2.1, which is not able to consume from 0.9.0.1 broker

Eef, ok so that won't help us (now anyway).

We might need to upgrade the main Kafka clusters before we proceed with the jumbo migration. Yarrrrrr

Ottomata moved this task from Incoming to Kafka Work on the Analytics board.Mar 29 2018, 4:45 PM
Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Apr 5 2018, 3:28 PM

Moving this to done. I think it is best to stick with MirrorMaker for now. We know why (mostly) it wasn't working, and expect it to work better in later versions, after we upgrade Kafka main.

Nuria closed this task as Resolved.Apr 12 2018, 10:07 PM
Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM