In T314160: Q1:rack/setup/install kafka-stretch200[12], we received and racked hardware for a multi DC Kafka stretch cluster.
T307944: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP capture much of the info needed to set this up. This task will track and serve as a parent task for the remaining work needed.
Discovery-Search may want to use this for T317045: [Epic] Re-architect the Search Update Pipeline, as it will be a way to produce large events in a multi DC Kafka cluster without messing with Kafka main.
- T307944: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP
- T314160: Q1:rack/setup/install kafka-stretch200[12]
- New debian package for latest Kafka version (T300102 ?)
- Node (ganeti?) in tiebreaker DC (ulsfo)
- Puppetization for new Kafka cluster (probably requires some refactoring)
- T282887: Avoid accepting Kafka messages with whacky timestamps
- Kafka MirrorMaker (likely 1, unless we want to set up and use MirrorMaker 2 as well) instance to mirror from new Kafka stretch cluster to Kafka jumbo
- Gobblin job to ingest into HDFS
- ... ?
There will be decisions to make along the way, e.g.
- Name of new cluster? is stretch sufficient? Might be better to come up with something that doesn't refer to the replication/architecture
- topic prefix naming convention?
- How does this interact with[[ https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Hadoop_Event_Ingestion_Lifecycle | Hadoop Ingestion ]]?
- ...?