Upgrade to Kafka MirrorMaker 2 and revisit Kafka topic prefix convention
Open, LowPublic
Actions

Assigned To

None

Authored By

	Ottomata
	Mar 15 2021, 1:43 PM

Description

MirrorMaker 1 will likely be deprecated in the upcoming Kafka 3.0.0 release:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-720%3A+Deprecate+MirrorMaker+v1

MirrorMaker 2 has the ability to auto-prefix mirrored topics. This would allow us to maintain our active/active mirroring topic layout, while making producers Kafka topic prefix unaware. This could add some complexity, so we should think about it a little more first.

While we are at it, we should reconsider the use of 'data center' as the Kafka topic prefix.

See: https://wikitech.wikimedia.org/wiki/Kafka#Data_center_topic_prefixing_design_flaw

Related Objects

Mentioned In: T304373: Also intake Network Error Logging events into the Analytics Data Lake
T382065: Add support for active/active double compute streams in the EventStreams HTTP service

Event Timeline

Ottomata created this task.Mar 15 2021, 1:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 15 2021, 1:43 PM

Ottomata triaged this task as Low priority.Mar 15 2021, 1:43 PM

• fdans edited projects, added Analytics-Radar; removed Analytics.Mar 18 2021, 4:47 PM

Aklapper added a project: Data-Engineering-Icebox-DEPRECATED.Feb 10 2023, 5:44 PM

I don't see a clear action for SRE so I'm removing the tag. Please add us back when needed.

BTullis edited projects, added Data-Platform-SRE; removed Analytics-Radar.Jul 11 2023, 9:30 PM

Gehel moved this task from Incoming to Software Upgrades on the Data-Platform-SRE board.Dec 7 2023, 2:16 PM

Ottomata renamed this task from Upgrade to Kafka MirrorMaker 2 to Upgrade to Kafka MirrorMaker 2 and revisit Kafka topic prefix convention.May 8 2024, 3:22 PM

Ottomata updated the task description. (Show Details)

Ottomata added a project: Event-Platform.

Ottomata edited subscribers, added: gmodena, tchin; removed: • razzi.

Maintenance_bot edited projects, added Data-Engineering; removed Data-Engineering-Icebox-DEPRECATED.May 8 2024, 3:31 PM

• lbowmaker moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.May 17 2024, 11:31 AM

Ottomata added a subscriber: brouberol.Jul 23 2024, 2:38 PM

Ottomata mentioned this in T382065: Add support for active/active double compute streams in the EventStreams HTTP service.Dec 13 2024, 3:04 PM

• VirginiaPoundstone added a project: Data-Engineering-Radar.Jan 6 2025, 9:14 PM

Ottomata moved this task from Backlog to Components on the Event-Platform board.Sep 17 2025, 3:03 PM

Ottomata mentioned this in T304373: Also intake Network Error Logging events into the Analytics Data Lake.Sep 29 2025, 4:00 PM

JMonton-WMF subscribed.Mon, Feb 2, 9:44 AM