Page MenuHomePhabricator

Switch to --new.consumer for main -> analytics MirrorMaker
Closed, ResolvedPublic5 Story Points

Description

This will ease the Kafka cluster upgrade(s), as we can run a 0.9 MirrorMaker using --new.consumer to consume from a 1.x cluster and produce to a 0.9 cluster. We can then upgrade Kafka main and get a stable 1.x MirrorMaker for main -> jumbo before proceeding with the analytics to jumbo Kafka client migrations.

This will also allow us to remove old graphite based MirrorMaker dashboards and metrics.

Event Timeline

Ottomata triaged this task as Normal priority.Apr 17 2018, 5:59 PM
Ottomata created this task.
Ottomata moved this task from Incoming to Kafka Work on the Analytics board.

Change 427400 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Split role::kafka::analytics into different roles so we can apply different hiera

https://gerrit.wikimedia.org/r/427400

Change 427400 merged by Ottomata:
[operations/puppet@production] Split role::kafka::analytics into 2 roles so we can apply different hiera

https://gerrit.wikimedia.org/r/427400

Change 427406 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use --new.consumer for main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/427406

Mentioned in SAL (#wikimedia-operations) [2018-04-23T14:10:19Z] <ottomata> switching main -> analytics MirrorMaker to --new.consumer (temporarily stopping puppet on kafka101[234]) https://phabricator.wikimedia.org/T192387

Mentioned in SAL (#wikimedia-analytics) [2018-04-23T14:10:24Z] <ottomata> switching main -> analytics MirrorMaker to --new.consumer (temporarily stopping puppet on kafka101[234]) https://phabricator.wikimedia.org/T192387

Change 427406 merged by Ottomata:
[operations/puppet@production] Use --new.consumer for main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/427406

Change 428350 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add prometheus based alerts for main-eqiad -> analytics eqiad

https://gerrit.wikimedia.org/r/428350

Change 428350 merged by Ottomata:
[operations/puppet@production] Add prometheus based alerts for main-eqiad -> analytics eqiad

https://gerrit.wikimedia.org/r/428350

Nuria closed this task as Resolved.Jun 11 2018, 11:05 PM