Page MenuHomePhabricator

EventGate should extract event time from events and produce to kafka with timestamp
Closed, ResolvedPublic3 Story Points

Description

Add a new config to wikimedia-eventgate (and/or default-eventgate?) to optionally allow for extraction of event time from events, and then use this event time as the Kafka message timestamp when producing to Kafka.

Event Timeline

Ottomata created this task.Mar 28 2019, 3:11 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMar 28 2019, 3:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Milimetric triaged this task as High priority.Apr 1 2019, 3:52 PM
Ottomata moved this task from Backlog to In Progress on the EventBus board.Apr 8 2019, 8:11 PM

@Pchelolo q for ya. Once we start using dt for Kafka message timestamp, the test/event examples field that I'm using for k8s readinessProbe send the same event everytime with the same dt. I think we should set dt to current time every time we post that event, but I'm not sure of the best way to do this. I suppose I could change the exec so it parses the examples and modifies the dt. Would be nice to be able do to this with CLI tools, but first I'd need something like yq in the docker image (just made T220509).

Hm, I went ahead and modified scripts/post-events to support a --dt-field option. If given, the script will set the provided dt field to a current ISO-8601 string. The downside is this assumes that events have ISO-8601 event times (like we do), but it is optional anyway.

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Apr 9 2019, 4:02 PM

Change 502534 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Use new readinessProbe post-events flag to set meta.dt

https://gerrit.wikimedia.org/r/502534

Change 502534 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Use new readinessProbe post-events flag to set meta.dt

https://gerrit.wikimedia.org/r/502534

Change 502539 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Use meta.dt as dt_field for Kafka message timestamp

https://gerrit.wikimedia.org/r/502539

Change 502539 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Use meta.dt as dt_field for Kafka message timestamp

https://gerrit.wikimedia.org/r/502539

Ottomata set the point value for this task to 3.Apr 10 2019, 4:01 PM
Ottomata moved this task from In Progress to Done on the EventBus board.Apr 19 2019, 4:17 PM
Nuria closed this task as Resolved.May 14 2019, 9:32 PM