librdkafka 0.9.4 supports producing to Kafka with a Kafka message event timestamp. varnishkafka should be configurable so that it can select a field out of the varnishlog data to use as the event timestamp when producing to Kafka.
This is pretty low priority, as the difference in request and server side produce time should usually be very small, but it would be nice to have this set for all our Kafka data if we can.
To set the kafka metadata timestamp to the data timestamp so you can use time based consumption, producer adds timestamp and broker uses that time as the kafka message
Before even starting it would be awesome to merge https://gerrit.wikimedia.org/r/#/c/421490 to have some sanity in formatting.
Useful info: from https://github.com/edenhill/librdkafka/issues/1016 it seems there is a new api from 0.9.4 able to set timestamp: https://github.com/edenhill/librdkafka/commit/15d3e7ea8589fcf0a81f5631e9d74f88630fa7bf
@colewhite since you are poking around in the varnishkafka code...if you feel like having some fun this could be interesting!
Very low priority tho, so don't worry about it unless you want to. :)
We are moving to ATS, I'll add this requirement for the tool that will replace Varnishkafka.