Page MenuHomePhabricator

EventGate should extract event time from events and produce to kafka with timestamp
Closed, ResolvedPublic3 Estimated Story Points

Description

Add a new config to wikimedia-eventgate (and/or default-eventgate?) to optionally allow for extraction of event time from events, and then use this event time as the Kafka message timestamp when producing to Kafka.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@Pchelolo q for ya. Once we start using dt for Kafka message timestamp, the test/event examples field that I'm using for k8s readinessProbe send the same event everytime with the same dt. I think we should set dt to current time every time we post that event, but I'm not sure of the best way to do this. I suppose I could change the exec so it parses the examples and modifies the dt. Would be nice to be able do to this with CLI tools, but first I'd need something like yq in the docker image (just made T220509).

Hm, I went ahead and modified scripts/post-events to support a --dt-field option. If given, the script will set the provided dt field to a current ISO-8601 string. The downside is this assumes that events have ISO-8601 event times (like we do), but it is optional anyway.

Change 502534 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Use new readinessProbe post-events flag to set meta.dt

https://gerrit.wikimedia.org/r/502534

Change 502534 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Use new readinessProbe post-events flag to set meta.dt

https://gerrit.wikimedia.org/r/502534

Change 502539 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Use meta.dt as dt_field for Kafka message timestamp

https://gerrit.wikimedia.org/r/502539

Change 502539 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Use meta.dt as dt_field for Kafka message timestamp

https://gerrit.wikimedia.org/r/502539

Ottomata set the point value for this task to 3.Apr 10 2019, 4:01 PM