I'd say this is medium to low priority and is something that needs to be worked on in collaboration with maintainers of other Kafka clusters.

This happened today, somehow there were recentchange events with timestamps from around 2007 in the kafka stream.

Happened again today. There was a mediawiki.recentchange event with a 2015 timestamp.

This is a nasty bug if Andrew happens to not be around, I just wanna ++ the tech debt value here.

Relevant:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation

There are 2 possible scenarios causing different issues:

• Future timestamps that can block the Kafka's clean-up policy, filling disks.
• Future or Past timestamps that could cause issues in Data processing pipelines.
  
  For the first one, it happens because the clean-up policy "delete" in Kafka only checks the first offset, and if it has passed the retention period, it is removed. If the first offset has a wrong date, for example, in the future, Kafka won't continue looking for other messages to be removed, and the topic will keep all the data until the "future date + retention" is reached, which can fill the disks.
  
  These 2 tickets https://phabricator.wikimedia.org/T267648, https://phabricator.wikimedia.org/T376026 already solved the biggest problem by setting the meta.dt manually rather than allowing clients to set any date.
  
  As commented here, reducing log.message.timestamp.difference.max.ms at broker level can cause issues in log compacted topics.
  
  Starting from Kafka v3.6.0, there is a new config (log.message.timestamp.after.max.ms) that can be set only for future events, which would solve the issue without affecting compacted topics.
  
  I guess upgrading from v1.1.0 to v3.6.0 is not a quick or simple solution for now.
  
  As an improvement, we can change the config log.message.timestamp.difference.max.ms in the topics created from logs, which are udp_localhost-info, udp_localhost-debug, udp_localhost-warning and, udp_localhost-err. Their cleanup.policy is "delete" and messages don't have keys in these topics, so they never will be compacted. And due to the nature of logs, there is no point on changing the policy to cleanup.policy "compact".
  
  This could be applied to other topics, although if Data pipelines are having issues with old timestamps, we might want to review each case as replying old messages could be a valid use case for data fixes. But I can apply the same config to some topics.
  
  @Ottomata, I'm not sure who can confirm this, it looks in general that this won't happen as the applications writing meta.dt are fixed, but I can change log.message.timestamp.difference.max.ms in the udp_localhost-* topics manually, so it won't happen on those in any case.

After some conversations, we have decide to not continue with this ticket until the cluster is upgraded to version >=3.6.0. (https://phabricator.wikimedia.org/T300102)

At that version of the cluster, we'll set message.timestamp.after.max.ms to a desired value and the cluster will reject future messages exceeding that value.