Currently EventGate and eventutilities libraries set event meta.dt timestamp field only if event producers don't already set them. This leads to some events being late in regard to their meta.dt timestamp, while they actually arrived 'in time' in terms on ingestion timestamp.
We wish to change the behavior of EventGate and the event-utilities libraries to always set meta.dt, overwriting the event producer provided value.
We must consider the canary events special case. Canary events are used by the platform to ensure that streams are working as expecting, and they are used by Hive ingestion to close hourly partition windows for streams that have no events in an hour.
The logic needed by event producing libraries is:
if meta.domain != 'canary':
meta.dt = current_date_timePlan:
- Update and deploy EventGate code - MR14 - Always overwrite meta.dt field with current date time, except for canary events
- Update event-utilities code - Always set meta.dt unless canary event (1172094)
- Update documentation
- Communicate to users: https://wikimedia.slack.com/archives/CSV483812/p1756305984271019