Grooming: we are going to put this in DE radar and hope that Search can fix this soonish. It isn't super urgent from our side (it is just causing logspam), but it does mean we can't unblock this log from logstash ingestion until this is fixed.

@dcausse do you have a sense of when you all might find time to work on this?

In T405949#11234025, @Ottomata wrote:

Grooming: we are going to put this in DE radar and hope that Search can fix this soonish. It isn't super urgent from our side (it is just causing logspam), but it does mean we can't unblock this log from logstash ingestion until this is fixed.

@dcausse do you have a sense of when you all might find time to work on this?

I think it's up to @gmodena and @BTracy-WMF to prioritize this work now but please let us how bad this is on your end. If really bad we might perhaps want to do a first step to stop emitting meta.dt and revisit later if we want to get more precise metrics in sparql query logs.

Nothing bad is happening on our end, except that we can't remove o11y's logstash filter for this logspam until we've silenced the known offenders :)

@BTracy-WMF @gmodena would love to resolve the tech debt on our side before we totally forget about it! Would you be able to put a change to QueryEventGenerator on a timeline somewhere? Still no hurry, but would be nice to know approx when it could be done.

In T405949#11715205, @Ottomata wrote:

@BTracy-WMF @gmodena would love to resolve the tech debt on our side before we totally forget about it! Would you be able to put a change to QueryEventGenerator on a timeline somewhere? Still no hurry, but would be nice to know approx when it could be done.

Hey @Ottomata,

I remember we did some analysis when we introduced a similar change in other pipelines (was it webrequest?); but do you have a sense of how much timestamp drift this new change will introduce?

My preference would be for WDQS to behave like all other EventPlatform producers and implement T376026: Update event-producing tools to overwrite `meta.dt`. Our next gen service will be compliant from the start, for this legacy codebase (that is EOL) I would need to understand the magnitude of the change. We would prefer to avoid touching Blazegraph
unless really necessary.

T431480 introduced the dt field for WDQSv2.

Nice!

It looks like I missed Gabriele's previous question! Sorry about that!

do you have a sense of how much timestamp drift this new change will introduce?

I think not much. As long as your app doesn't have any delay between when the event is created and POSTed to EventGate, the drift will be as much time as it takes for EventGate to receive and set meta.dt. I wouldn't expect more than 10s of milliseconds max.

We would prefer to avoid touching Blazegraph unless really necessary.

Makes sense. I think the priority here is low; it is just removing some logstash tech debt. Let us know when this BlazeGraph query logging is gone, and we can proceed!

In T405949#12138131, @Ottomata wrote:

Nice!

It looks like I missed Gabriele's previous question! Sorry about that!

do you have a sense of how much timestamp drift this new change will introduce?

I think not much. As long as your app doesn't have any delay between when the event is created and POSTed to EventGate, the drift will be as much time as it takes for EventGate to receive and set meta.dt. I wouldn't expect more than 10s of milliseconds max.

Let me discuss this internally with @BTracy-WMF and a couple of stakeholders, but the blast radius of this change seems minimal and restricted to (controlled) Wikidata Platform use cases.

This has been addressed for WDQS v2. Safe to resolve if it's acceptable from the Search pov