Change Details

This task tracks setting up a Kafka cluster dedicated to logging purposes. Given the considerations in {T205873} said cluster will initially coexist on Logstash hosts now and on dedicated hardware in the future. Outline of steps: [x] Puppetization for Kafka brokers, in a dedicated profile [x] Setup TLS certificates for the brokers [x] Deploy Kafka brokers to logstash elasticsearch data hosts [] Decide on the granularity of topics (one single topic? one per cluster?) and the number of partitions [] Setup Logstash input for Kafka logging (i.e. consumers) with TLS [] Setup rsyslog output for Kafka logging (i.e. producers) with TLS and make it opt-in

This task tracks setting up a Kafka cluster dedicated to logging purposes. Given the considerations in {T205873} said cluster will initially coexist on Logstash hosts now and on dedicated hardware in the future. Outline of steps: [x] Puppetization for Kafka brokers, in a dedicated profile [x] Setup TLS certificates for the brokers [x] Deploy Kafka brokers to logstash elasticsearch data hosts [x] Decide on the granularity of topics (one single topic? one per cluster?) and the number of partitions [] Setup Logstash input for Kafka logging (i.e. consumers) with TLS [x] Setup rsyslog output for Kafka logging (i.e. producers) with TLS and make it opt-in

This task tracks setting up a Kafka cluster dedicated to logging purposes. Given the considerations in {T205873} said cluster will initially coexist on Logstash hosts now and on dedicated hardware in the future. Outline of steps: [x] Puppetization for Kafka brokers, in a dedicated profile [x] Setup TLS certificates for the brokers [x] Deploy Kafka brokers to logstash elasticsearch data hosts [x] Decide on the granularity of topics (one single topic? one per cluster?) and the number of partitions [] Setup Logstash input for Kafka logging (i.e. consumers) with TLS [x] Setup rsyslog output for Kafka logging (i.e. producers) with TLS and make it opt-in