Page MenuHomePhabricator

Produce webrequests from varnishkafka to Kafka with Kafka message timestamp set to configurable content field
Closed, DeclinedPublic

Description

librdkafka 0.9.4 supports producing to Kafka with a Kafka message event timestamp. varnishkafka should be configurable so that it can select a field out of the varnishlog data to use as the event timestamp when producing to Kafka.

This is pretty low priority, as the difference in request and server side produce time should usually be very small, but it would be nice to have this set for all our Kafka data if we can.

Related Objects

StatusSubtypeAssignedTask
Declinedelukey
ResolvedOttomata
Resolvedmforns
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DuplicateNone
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
Resolvedelukey
Resolvedelukey
Resolvedelukey
ResolvedCmjohnson
ResolvedRobH
ResolvedOttomata
DeclinedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DeclinedNone
Resolveddcausse
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DuplicateOttomata
Resolvedelukey
ResolvedOttomata
Resolvedelukey
ResolvedOttomata
ResolvedVgutierrez
ResolvedOttomata
Resolvedelukey
Resolvedelukey

Event Timeline

Ottomata created this task.Jun 1 2017, 8:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 1 2017, 8:18 PM
Nuria moved this task from Incoming to Dashiki on the Analytics board.Jun 5 2017, 3:43 PM
Nuria added a subscriber: Nuria.Jul 10 2017, 4:00 PM

To set the kafka metadata timestamp to the data timestamp so you can use time based consumption, producer adds timestamp and broker uses that time as the kafka message

Nuria moved this task from Dashiki to Radar on the Analytics board.Jul 10 2017, 4:03 PM
Ottomata moved this task from Q1 2020/2021 to Backlog on the Analytics-Clusters board.
elukey claimed this task.Aug 4 2017, 9:48 AM
elukey added a project: User-Elukey.
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Aug 4 2017, 3:09 PM

Before even starting it would be awesome to merge https://gerrit.wikimedia.org/r/#/c/421490 to have some sanity in formatting.

Useful info: from https://github.com/edenhill/librdkafka/issues/1016 it seems there is a new api from 0.9.4 able to set timestamp: https://github.com/edenhill/librdkafka/commit/15d3e7ea8589fcf0a81f5631e9d74f88630fa7bf

elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Mar 23 2018, 3:54 PM
elukey moved this task from Analytics Backlog to Backlog on the User-Elukey board.Jan 7 2019, 8:21 AM

@colewhite since you are poking around in the varnishkafka code...if you feel like having some fun this could be interesting!

Very low priority tho, so don't worry about it unless you want to. :)

elukey closed this task as Declined.Jan 3 2020, 11:01 AM

We are moving to ATS, I'll add this requirement for the tool that will replace Varnishkafka.

Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM