/var/log/kafka/kafkaServer-gc.log is not properly rotated on kafka nodes. The root partition of kafka1020 and kafka1020 was full, leading to critical icinga alerts.

After investigating the issue with _joe_ we discovered that a few gc-related logging options could be used to rotate such files properly.

I have thus added KAFKA_OPTS="-XX:GCLogFileSize=1M" to /etc/default/kafka on kafka1020, restared the kafka service, and confirmed that kafkaServer-gc.log got properly truncated, solving the disk space issue.

After a few minutes I've added KAFKA_OPTS="-XX:GCLogFileSize=50M -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5" to /etc/default/kafka on kafka1012 to fix the same issue and also test proper log rotation rather than simply truncating the file. I've then restarted the kafka service on kafka1012 as well, observed that a new file called kafkaServer-gc.log.0.current was created and kafkaServer-gc.log was still on disk filling up the root partition. After checking that kafkaServer-gc.log was not used by the java process, I've truncated it to reclaim disk space.

See the following code reviews for the details of pending puppet changes:
https://gerrit.wikimedia.org/r/#/c/266203/
https://gerrit.wikimedia.org/r/#/c/266209/

The puppet changes have been merged. I have re-enabled puppet on kafka1020 and kafka1012.

On kafka1018, where the size of kafkaServer-gc.log was also starting to become a problem, I have restarted kafka after puppet modified /etc/default/kafka and truncated kafkaServer-gc.log. @JAllemandou restarted eventlogging shortly afterwards.

1014, 1018 and 1022 Kafka brokers were restarted and cleaned from the huge logs.

Last action was to run kafka preferred-replica-election to rebalance the partition leaders.

EL was restarted again.