Does some kind of similar logging/tracking already exist in Query Service? What information does it contain?

WDQS has some facilities to log the SPARQL queries it receives. The logs contain the information as described in this schema: https://schema.wikimedia.org/repositories//secondary/jsonschema/sparql/query/current.yaml

Based on your impression, how feasible to you think such initiative is (especially for a team that is not yet familiar with the code base)? Do you immediately see some potential pitfalls?

Query logging supports two outputs:

• event-platform which then allows to have this data in our data-lake (hadoop)
• file, enabled by adding the following JVM arguments to blazegraph: -Dwdqs.event-sender-filter.file-event-sender=true -DDwdqs.event-sender-filter.file-event-sender-filepath=/path/to/logged_queries.json

Can Query Service backend recognize when a query it is supposed to execute started on its own UI vs. it is a federated query that started on some other SPARQL endpoint?

Knowing the provenance of a query is rather difficult, I could see three main cases:

• A human using the WDQS UI
• A automated client using the sparql endpoint directly
• Another sparql engine using the sparql endpoint to run a federated query

I doubt that distinguishing between these 3 is going to be trivial and some research might have to be done here.

User-Agent might be usable if they're well-known. You can tune the UA that wdqs will use to run federated queries using:

• -Dcom.bigdata.rdf.sail.webapp.client.HttpClientConfigurator=org.wikidata.query.rdf.blazegraph.ProxiedHttpConnectionFactory
• -Dhttp.userAgent="A custom UA"

Is there a sure way to differentiate between situations when SERVICE is used to call an external endpoint (what we would consider a federated query) vs. used to call some helpers like label service or similar? Would it have to rely on some fixed list of exceptions? Or is there a more elegant way to do this?

So for analyzing outgoing traffic I would rely on logged sparql, it's very likely that you have an allowlist somewhere, if not it would be easy to exclude the service you know are "technical" (label service, custom blazegraph features, mwapi). Other techniques would involve relying on ProxiedHttpConnectionFactory and adapt it in some ways allow to emitting the metrics you need based on the requests it's running, I don't know how hard that would be.

In short:

• Monitoring outgoing federation is doable by analyzing the query logs
• Monitoring incoming federation is a lot harder but could be approximated from query logs too by analyzing well-known user-agents
  • Caveats: A single federated query can make several calls to the federated service which might make it even harder to get the real number, blazegraph does pass a unique queryId param which might help to de-duplicate those but I doubt this is standard.

Adding more instrumentation to wdqs/blazegraph is probably possible, this might only help to monitor outgoing federation but is unlikely to help to monitor incoming federation.
If the set of sparql endpoints is well-known they could possibly agree on some standards with, i.e. a recognizable UA or some other headers.

Thanks a lot for the clarification and the comments in the doc.
Our goal for this quarter is to establish a baseline for some kind of an (even imperfect) metric.
We decided to investigate 3 different opportunities:

• measuring federation on WDQS/WBQS UI - T394988
• measuring federation on WDQS/WBQS backend - T395044
• if we don't get a quick win with any of the above, we want to at least report on a super basic metric like 'the average size of the Allowlist' of known active instances in the ecosystem. It's far from ideal, but this is a backup plan to meet our key result this quarter.

We have some additional questions and clarifications and would appreciate any help there.

1. Would it be possible on your side to implement a simple API for the WDQS/WBQS backend that would return the contents of the Allowlist? Or can we go ahead and implement it? It would be useful for us for different reasons:
   1. To measure our 'backup' metric in an automated way
   2. To differentiate between the use of the SERVICE syntax for real federation vs. helpers in the 'measuring on UI' strategy
   3. In general, for future improvements of UX with WBQS/WDQS UI to let the users know where they can federate with and pre-populate prefixes

2. Could you please confirm whether our understanding described in the paragraph below is correct? >>>> If we analyze query logs on any given QS backend, we will observe both 'original' queries (those that were received from UI or automated client) & subqueries executed on behalf of another SPARQL endpoint as part of a federated query. It's difficult to recognize them from each other, but the use of User-Agents could be helpful there. When executing a subquery, the endpoint will not see the original full query it was extracted from. It would only see the query that was created by the coordinating endpoint that sent this query.

The second point turned out relevant not just for the work we plan to do for federation, but also for the work that another team at WMDE is doing (cc @Ifrahkhanyaree_WMDE @AndrewTavis_WMDE ): they are analyzing queries to understand the most popular queries that users run on Wikidata QS. I realized that the results of the analysis might be distorted by the subqueries, cause they were not actually written by users, but appeared from incoming federation to Wikidata.

3. It's possible that I already asked this question during the call, so I apologize if you already answered it. Do you have an idea whether User-Agents could be useful if we wanted to differentiate between queries coming from the QS query editor vs. the Query Builder?

1. Would it be possible on your side to implement a simple API for the WDQS/WBQS backend that would return the contents of the Allowlist? Or can we go ahead and implement it? It would be useful for us for different reasons:

It should be possible and not too difficult to expose this list through an HTTP call somewhere. Ideally, could you take care of the implementation? We can help review, merge and deploy.

2. Could you please confirm whether our understanding described in the paragraph below is correct? >>>> If we analyze query logs on any given QS backend, we will observe both 'original' queries (those that were received from UI or automated client) & subqueries executed on behalf of another SPARQL endpoint as part of a federated query. It's difficult to recognize them from each other, but the use of User-Agents could be helpful there. When executing a subquery, the endpoint will not see the original full query it was extracted from. It would only see the query that was created by the coordinating endpoint that sent this query.

The second point turned out relevant not just for the work we plan to do for federation, but also for the work that another team at WMDE is doing (cc @Ifrahkhanyaree_WMDE @AndrewTavis_WMDE ): they are analyzing queries to understand the most popular queries that users run on Wikidata QS. I realized that the results of the analysis might be distorted by the subqueries, cause they were not actually written by users, but appeared from incoming federation to Wikidata.

This all seems to match my understanding.

3. It's possible that I already asked this question during the call, so I apologize if you already answered it. Do you have an idea whether User-Agents could be useful if we wanted to differentiate between queries coming from the QS query editor vs. the Query Builder?

In both cases, the User-Agent should be the user's browser, so we can't use it to differentiate. It might be possible to add instrumentation to Query UI and Query Builder to add metadata (as comment to the query, or as HTTP headers). WMDE owns the UIs, so you might know better than we do.

Hi @Gehel !
Thanks again for all the clarifications :)
We have new questions, could you please have a look when you have an opportunity.

1. Correct me if I'm wrong - after the graph split, there are 2 Query Service instances running - one for Wikidata main and another for Wikidata scholarly articles. Does the data for both end up in the same dataset of query logs on hive?

2. While we started our first experiments with recognizing federated queries in that dataset T395044, we noticed that there is a huge amount of queries that have a malformed SERVICE line with no endpoint specified (so they end up self-federating and naturally all fail). We're now wondering whether these are legit queries that people write, or is this some artifact of the graph split, or incorrect logging/parsing. Do you maybe have some ideas where these can be coming from?

3. From what we know there should also be a Query Service running for Wikimedia Commons (not sure if it's being used though). Do you by any chance know whether it is also similarly instrumented to collect data somewhere?

Thank you!

CC @Tarrow

In T391383#10890326, @Anton.Kokh wrote:

Hi @Gehel !
Thanks again for all the clarifications :)
We have new questions, could you please have a look when you have an opportunity.

1. Correct me if I'm wrong - after the graph split, there are 2 Query Service instances running - one for Wikidata main and another for Wikidata scholarly articles. Does the data for both end up in the same dataset of query logs on hive?

This is essentially defined by the profile::query_service::sparql_query_stream hieradata in the puppet repo. This directs blazegraph to log to the named stream, which is then picked up by the generic event import into hive. The events should all have a graph_name column that distinguishes which graph the event came from.

2. While we started our first experiments with recognizing federated queries in that dataset T395044, we noticed that there is a huge amount of queries that have a malformed SERVICE line with no endpoint specified (so they end up self-federating and naturally all fail). We're now wondering whether these are legit queries that people write, or is this some artifact of the graph split, or incorrect logging/parsing. Do you maybe have some ideas where these can be coming from?

Not particularly sure. If you have some example queries maybe we can see if they generate any particular logging from the backend when they fail, and check how often the backend has similar logs? As a publicly exposed service it's actually quite rare for us to understand what end users are doing or where they are coming from.

3. From what we know there should also be a Query Service running for Wikimedia Commons (not sure if it's being used though). Do you by any chance know whether it is also similarly instrumented to collect data somewhere?

I see the logs making it into kafka, but not hive. I put together https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1154300 which we can ship next week which will turn on the hive ingestion.

Thank you!

CC @Tarrow

Change #1154300 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] EventStream: Enable hive ingestion for wcqs-external.sparql-query

https://gerrit.wikimedia.org/r/1154300

Change #1154300 merged by jenkins-bot:

[operations/mediawiki-config@master] EventStream: Enable hive ingestion for wcqs-external.sparql-query

https://gerrit.wikimedia.org/r/1154300

Mentioned in SAL (#wikimedia-operations) [2025-08-25T20:18:31Z] <ebernhardson@deploy1003> Started scap sync-world: Backport for [[gerrit:1154300|EventStream: Enable hive ingestion for wcqs-external.sparql-query (T391383)]], [[gerrit:1180610|cirrus: Enable phrase suggester variant (T397083)]]

Mentioned in SAL (#wikimedia-operations) [2025-08-25T20:23:53Z] <ebernhardson@deploy1003> ebernhardson: Backport for [[gerrit:1154300|EventStream: Enable hive ingestion for wcqs-external.sparql-query (T391383)]], [[gerrit:1180610|cirrus: Enable phrase suggester variant (T397083)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-08-25T20:31:36Z] <ebernhardson@deploy1003> Finished scap sync-world: Backport for [[gerrit:1154300|EventStream: Enable hive ingestion for wcqs-external.sparql-query (T391383)]], [[gerrit:1180610|cirrus: Enable phrase suggester variant (T397083)]] (duration: 13m 04s)

Appologies for taking so long, i noticed this patch in gerrit recently and apparently I forgot to previously deploy it.