Maniphest T190049

Spike: Consider alternatives to MirrorMaker: uReplicator, Confluent Replicator
Closed, ResolvedPublic13 Estimated Story Points
Actions

Assigned To

Authored By

	Ottomata
	Mar 19 2018, 2:31 PM

Description

MirrorMaker has always been a little flaky (requiring manual restarts to get proper partition rebalance assignment, etc.), but is being especially so when replicating from the 0.9 main Kafka cluster to the new 1.0 jumbo Kafka cluster. Other's have also experienced this MirrorMaker flakiness. We should try out other options and consider switching, especially since T189464 now blocks the Kafka jumbo migration tasks.

Uber's uReplicator
- open source
- Kafka 0.10 client (should be ok)

Confluent Replicator
- Closed sourced, but still free
- Just a Kafka Connect connector
- Latest Kafka, backed by Confluent

Related Objects
Search...

Status	Assigned	Task
Declined	elukey	T166833 Produce webrequests from varnishkafka to Kafka with Kafka message timestamp set to configurable content field
Resolved	Ottomata	T152015 Provision new Kafka cluster(s) with security features
Resolved	Lucas_Werkmeister_WMDE	T145712 Use RDF statement counts from entity data, not page props ( wikibase:identifiers, wikibase:statements and wikibase:sitelinks )
Resolved	Ottomata	T161731 Create reliable change stream for specific wiki
Resolved	Ottomata	T183303 Decomission old analytics kafka cluster
Resolved	Ottomata	T175461 Port Kafka clients to new jumbo cluster
Resolved	Gehel	T189458 re-enable wdqs kafka poller
Resolved	Ottomata	T189464 Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo
Resolved	Ottomata	T190049 Spike: Consider alternatives to MirrorMaker: uReplicator, Confluent Replicator

Event Timeline

Ottomata triaged this task as High priority.Mar 19 2018, 2:31 PM

Ottomata created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 19 2018, 2:31 PM

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Mar 19 2018, 2:31 PM

From https://github.com/uber/uReplicator/issues/8#issuecomment-375104514

uReplicator consumer is using 0.10.2.1, which is not able to consume from 0.9.0.1 broker

Eef, ok so that won't help us (now anyway).

We might need to upgrade the main Kafka clusters before we proceed with the jumbo migration. Yarrrrrr

Ottomata moved this task from Incoming to Kafka Work on the Analytics board.Mar 29 2018, 4:45 PM

Moving this to done. I think it is best to stick with MirrorMaker for now. We know why (mostly) it wasn't working, and expect it to work better in later versions, after we upgrade Kafka main.

Nuria closed this task as Resolved.Apr 12 2018, 10:07 PM

Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM

Spike: Consider alternatives to MirrorMaker: uReplicator, Confluent ReplicatorClosed, ResolvedPublic13 Estimated Story PointsActions

Description

Related ObjectsSearch...

Event Timeline

Spike: Consider alternatives to MirrorMaker: uReplicator, Confluent Replicator
Closed, ResolvedPublic13 Estimated Story Points
Actions

Related Objects
Search...