Now that main Kafka clusters are on Kafka 1.x, we can take advantage of Kafka's timestamp indexes. We should produce the Kafka message with the timestamp set to the event's meta.dt field.
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Ottomata | T196009 Support timestamp based consumption in KafkaSSE and EventStreams | |||
| Resolved | Ottomata | T196407 EventBus should produce messages to Kafka with event time set to meta.dt |
Event Timeline
Comment Actions
Change 437479 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventlogging@master] Produce kafka-python messages with timestamp set to Event time
Comment Actions
Ah, we need to change a Kafka broker setting to support this too. log.message.timestamp.type should be set to CreateTime, rather than LogAppendTime. Need to understand what happens to records that don't have event timestamps set if we are using CreateTime.
See also: https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message
Comment Actions
Change 437614 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Kafka - Change default message.timestamp.type to CreateTime
Comment Actions
Change 437614 merged by Ottomata:
[operations/puppet@production] Kafka - Change default message.timestamp.type to CreateTime
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2018-06-07T14:55:59Z] <ottomata> beginning rolling restarts of all cluster kafka brokers to apply log.message.timestamp.type=CreateTime - T196407
Comment Actions
Change 437479 merged by Ottomata:
[eventlogging@master] Produce kafka-python messages with timestamp set to Event time
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2018-06-11T13:52:55Z] <otto@deploy1001> Started deploy [eventlogging/eventbus@08a1dff]: Producing events with kafka timestamp set to event time - T196407
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2018-06-11T13:54:50Z] <otto@deploy1001> Finished deploy [eventlogging/eventbus@08a1dff]: Producing events with kafka timestamp set to event time - T196407 (duration: 01m 55s)
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2018-06-11T13:56:24Z] <otto@deploy1001> Started deploy [eventlogging/analytics@08a1dff]: Producing events with kafka timestamp set to event time - T196407
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2018-06-11T13:56:28Z] <otto@deploy1001> Finished deploy [eventlogging/analytics@08a1dff]: Producing events with kafka timestamp set to event time - T196407 (duration: 00m 04s)