We observed a weird Kafka behavior since we upgraded to 0.9: right after a broker restart the data log mtime gets reset to now. This messes up a lot with time based log retention, since the Kafka log cleaner uses the data log inode's mtime value to establish when a log file on disk needs to be cleaned.
We sent an email to kafka-users@ tracked in this email thread
One user pointed us to the following upstream bug: https://issues.apache.org/jira/browse/KAFKA-1379
The only workaround proposed, for the moment, is to set the retention.bytes configure option to control the maximum size of a topic partition. We are currently using a similar trick but with retention.ms:
This phab task is meant to track upstream changes and to decide how we should proceed from now with Kafka restarts.