Change Details

This kafka topic will be produced to once a week from an oozie workflow. It will be consumed by two separate consumer groups, one for each datacenter. Data size is estimated as ~3GB per week (with gzip compression enabled), but all generated at once. Kafka cluster: main-{eqiad,codfw} Topic name: cirrussearch_analytics_updates number of partitions: 2? 3? Not sure what makes sense, but the 3GB per week is going to be produced over a very short time. Additionally consumers will be running on ~35 hosts per cluster, so having a couple partitions so more than one can work through the queue would be convenient.

This kafka topic will be produced to once a week from an oozie workflow. It will be consumed by two separate consumer groups, one for each datacenter. Data size is estimated as ~3GB per week (with gzip compression enabled), but all generated at once. Kafka cluster: main-{eqiad,codfw} Topic name: eqiad.cirrussearch.analytics-updates Number of partitions: 3

This kafka topic will be produced to once a week from an oozie workflow. It will be consumed by two separate consumer groups, one for each datacenter. Data size is estimated as ~3GB per week (with gzip compression enabled), but all generated at once. Kafka cluster: main-{eqiad,codfw} Topic name: cirrussearch_analytics_updates number of partitions: 2? 3? Not sure what makes sense, but the 3GB per week is going to be produced over a very short time. Additionally consumers will be running on ~35 hosts per cluster, so having a couple partitions so more than one can work through the queue would be convenient.Topic name: eqiad.cirrussearch.analytics-updates Number of partitions: 3