Add a new config to wikimedia-eventgate (and/or default-eventgate?) to optionally allow for extraction of event time from events, and then use this event time as the Kafka message timestamp when producing to Kafka.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | Ottomata | T185233 Modern Event Platform | |||
Resolved | Ottomata | T201068 Modern Event Platform: Stream Intake Service | |||
Resolved | Ottomata | T206785 Modern Event Platform: Stream Intake Service (EventGate): Implementation | |||
Resolved | Ottomata | T219513 EventGate should extract event time from events and produce to kafka with timestamp |
Event Timeline
@Pchelolo q for ya. Once we start using dt for Kafka message timestamp, the test/event examples field that I'm using for k8s readinessProbe send the same event everytime with the same dt. I think we should set dt to current time every time we post that event, but I'm not sure of the best way to do this. I suppose I could change the exec so it parses the examples and modifies the dt. Would be nice to be able do to this with CLI tools, but first I'd need something like yq in the docker image (just made T220509).
Hm, I went ahead and modified scripts/post-events to support a --dt-field option. If given, the script will set the provided dt field to a current ISO-8601 string. The downside is this assumes that events have ISO-8601 event times (like we do), but it is optional anyway.
Change 502534 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Use new readinessProbe post-events flag to set meta.dt
Change 502534 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Use new readinessProbe post-events flag to set meta.dt
Change 502539 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Use meta.dt as dt_field for Kafka message timestamp
Change 502539 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Use meta.dt as dt_field for Kafka message timestamp