@Ladsgroup could you have a look at this and make sure it is correct and add the additional info you have? Thanks :)

Amir noticed that we might be tracking result views for the query builder incorrectly

@Ladsgroup any specific details?

Sorry I missed this. Let me explain:

Lots of grafana panels in query service ui dashboard solely look at wikibase.queryService.ui.app... For example rows served has wikibase.queryService.ui.app.result.resultLength. but the embed.js doesn't send rows returned to this endpoint. It sends it to wikibase.queryService.ui.embed.app.result.resultLength (there's an "embed" between ui and app) meaning we are missing all data we have for cases serving from embed.js.

Right, so it is mainly fixes to the grafana dashboard rather than sending different / new data?

Depends on the eye of the beholder. If it's just a mistake, we can actually fix sending the data so both hit the same endpoint?

In T272128#6806414, @Ladsgroup wrote:

Depends on the eye of the beholder. If it's just a mistake, we can actually fix sending the data so both hit the same endpoint?

The data is deliberately separated so that it can be displayed separately.
I think the main (possibly only) example of this is the result views

I guess its a question for @Lydia_Pintscher regarding exactly what the dashboard should be showing.
Should the "rows served" show all rows (including via embeds) or is it meant to only show people actually interesting with the main part of the UI?

In T272128#6811081, @Addshore wrote:

Should the "rows served" show all rows (including via embeds) or is it meant to only show people actually interesting with the main part of the UI?

I would say it doesn't make sense to distinguish embedded and non-embedded views for rows served there so I'd include both in this graph.

In T272128#6811226, @Lydia_Pintscher wrote:

In T272128#6811081, @Addshore wrote:

Should the "rows served" show all rows (including via embeds) or is it meant to only show people actually interesting with the main part of the UI?

I would say it doesn't make sense to distinguish embedded and non-embedded views for rows served there so I'd include both in this graph.

Do you still want the split display for embedded vs non embedded result view types?

In T272128#6811447, @Addshore wrote:

Do you still want the split display for embedded vs non embedded result view types?

Yes that's pretty useful.

Description is ready to go now IMO

One note, it's not as easy as it looks because one of the paths has an extra node.
E.g. wikibase.queryService.ui.app.result.received.success.sum and wikibase.queryService.ui.embed.app.result.received.success.sum

And so far wikibase.queryService.ui*.app.result.received.success.sum hasn't worked, neither anything else I tried like wikibase.queryService.ui{.embed,}.app.result.received.success.sum or wikibase.queryService.ui.embed?.app.result.received.success.sum. The doc explains why: https://graphite.readthedocs.io/en/latest/terminology.html#term-series-list and https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards

The only way is to combine data in each series separately once they are processed which depends on what data is being extracted from.

group() would work here but sometimes doesn't, depending on how we are transforming data.

Would it be an option to change the tracking key and then move the existing data in Graphite? (Maybe literally mv them – IIRC they’re actually stored as files in directories?)

I spent more than two hours just trying to make this work. It's quite messy, at the end I just abandoned it. Let's try to move the data instead.

Change 704365 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[wikidata/query/gui@master] Make embed and non-embed tracking namespace have equal footing

https://gerrit.wikimedia.org/r/704365

Change 704365 merged by jenkins-bot:

[wikidata/query/gui@master] Make embed and non-embed tracking namespace have equal footing

https://gerrit.wikimedia.org/r/704365

Tomorrow all metrics starting with wikibase.queryService.ui.app. should be migrated to wikibase.queryService.ui.index.app. I will deploy it around 8:30 UTC

In T272128#7212524, @Ladsgroup wrote:

Tomorrow all metrics starting with wikibase.queryService.ui.app. should be migrated to wikibase.queryService.ui.index.app. I will deploy it around 8:30 UTC

For my reference at deploy time on the graphite hosts this translates to:

sudo -u _graphite -s /bin/bash
install -d /srv/carbon/whisper/wikibase/queryService/ui/index
cp -vr /srv/carbon/whisper/wikibase/queryService/ui/app /srv/carbon/whisper/wikibase/queryService/ui/index

We'll be copying instead of moving in case sth goes wrong we'd have the old data at least, old metrics can be then reclaimed/deleted

Change 704756 had a related patch set uploaded (by WDQSGuiBuilder; author: WDQSGuiBuilder):

[wikidata/query/gui-deploy@production] Merging from d84c67d9b71c6bc2586ced9412df8b839bf98318

https://gerrit.wikimedia.org/r/704756

Change 704756 merged by Ladsgroup:

[wikidata/query/gui-deploy@production] Merging from d84c67d9b71c6bc2586ced9412df8b839bf98318

https://gerrit.wikimedia.org/r/704756

Mentioned in SAL (#wikimedia-operations) [2021-07-15T10:41:54Z] <godog> move wikibase.queryService.ui.app to wikibase.queryService.ui.index.app - T272128

@Manuel I'll leave this for you to verify?