The timestamp set in Kafka by HAProxyKafka doesn't match with the timestamp set in the data dt field. This is not great because we want to use the Kafka timestamp for Gobblin to batch rows in hourly folders.
See discussions below, the problem can't be fixed at HAProxyKafka level for performance reasons so we fix it in Gobblin by parsing the data and use its dt field (same as currently done in webrequest).
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Update Gobblin webrequest_frontend timestamp | analytics/refinery | master | +4 -12 |
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Explicitly setting TimestampType as LogAppend | repos/sre/haproxykafka!81 | fabfur | ts-type | main | |
| Proposal: set kafka message timestamp to the content of `dt` field | repos/sre/haproxykafka!80 | fabfur | message-timestamp | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Antoine_Quhen | T354694 [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition | |||
| Resolved | JAllemandou | T387750 [HAProxy migration] Compile expected migration delta, switch over plan and communicate | |||
| Resolved | Fabfur | T351117 Move analytics log from Varnish to HAProxy | |||
| Resolved | Fabfur | T370668 New software: haproxykafka | |||
| Resolved | JAllemandou | T388397 Make webrequest_frontend being ingested using the in-data `dt` field | |||
| Resolved | Fabfur | T389521 Update HAProxyKafka kafka-timestamp type |
fabfur opened https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/80
Proposal: set kafka message timestamp to the content of dt field
Maybe I'm misreading the task description but from
If the problem can't be fixed at HAProxyKafka level, we'll fix it in Gobblin by parsing the data and use its dt field (same as currently done in webrequest).
I understand that for webrequest you're already using the dt field instead of the kafka message timestamp?
If that's right, why are you introducing this change in behavior?
You're right in your reading. Let me give some context and explanation.
When we started using Kafka for webrequest, the kafka timestamp feature was not existing, so we used (and still use) a parser to extract the timestamp from the data.
Now that the kafka-timestamp exists, we want to use it as much as we can to prevent parsing the data for the timestamp field only as the data-movement tool we use doesn't need to parse the JSON for anything else.
I answered to a comment on the gitlab PR (https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/80#note_131234) that I'll repeat here for consistency.
If there is a performance issue with getting the timestamp in HAProxyKafka, let's keep doing it in Gobblin where the performance gain/loss is not as important.
BTW, there was a request to do this for varnishkafka, but it was declined when it was intended to do it form ATS instead:
So, we should do it for haproxykafka too (if we can!)
fabfur opened https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/81
Explicitly setting TimestampType as LogAppend
fabfur merged https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/81
Explicitly setting TimestampType as LogAppend
Mentioned in SAL (#wikimedia-operations) [2025-03-20T16:46:28Z] <fabfur> imported haproxykafka 0.3.6 into apt repository (added TimestampType) (T388397)
Change #1129900 had a related patch set uploaded (by Joal; author: Joal):
[analytics/refinery@master] Update Gobblin webrequest_frontend timestamp
fabfur closed https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/80
Proposal: set kafka message timestamp to the content of dt field
Change #1129900 merged by Joal:
[analytics/refinery@master] Update Gobblin webrequest_frontend timestamp