Page MenuHomePhabricator

Produce webrequests from varnishkafka to Kafka with Kafka message timestamp set to configurable content field
Closed, DeclinedPublic

Description

librdkafka 0.9.4 supports producing to Kafka with a Kafka message event timestamp. varnishkafka should be configurable so that it can select a field out of the varnishlog data to use as the event timestamp when producing to Kafka.

This is pretty low priority, as the difference in request and server side produce time should usually be very small, but it would be nice to have this set for all our Kafka data if we can.

Related Objects

StatusSubtypeAssignedTask
Declined elukey
ResolvedOttomata
Resolvedmforns
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DuplicateNone
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
Resolved elukey
Resolved elukey
Resolved elukey
ResolvedCmjohnson
ResolvedRobH
ResolvedOttomata
DeclinedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DeclinedNone
Resolved dcausse
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DuplicateOttomata
Resolved elukey
ResolvedOttomata
Resolved elukey
ResolvedOttomata
ResolvedVgutierrez
ResolvedOttomata
Resolved elukey
Resolved elukey

Event Timeline

To set the kafka metadata timestamp to the data timestamp so you can use time based consumption, producer adds timestamp and broker uses that time as the kafka message

Before even starting it would be awesome to merge https://gerrit.wikimedia.org/r/#/c/421490 to have some sanity in formatting.

Useful info: from https://github.com/edenhill/librdkafka/issues/1016 it seems there is a new api from 0.9.4 able to set timestamp: https://github.com/edenhill/librdkafka/commit/15d3e7ea8589fcf0a81f5631e9d74f88630fa7bf

@colewhite since you are poking around in the varnishkafka code...if you feel like having some fun this could be interesting!

Very low priority tho, so don't worry about it unless you want to. :)

We are moving to ATS, I'll add this requirement for the tool that will replace Varnishkafka.