This is the highest priority investigation to get started with of the compass tickets.

I was reading through the code for fun while sipping a coffee, and saw some things of note that I figured I may as well mention here...

You'll likely want to take a look at the optimizers if you want to give this a go in the java code.
These are chained and parse and rewrite the SPARQL, and would be a good easy pluggable place therefore to add things that look at or alter said queries.
See blazegraph/src/main/java/org/wikidata/query/rdf/blazegraph/WikidataServicePlacementOptimizer.java for example that even looks at SERVICE nodes.
Probably then simply logging in a structured way would give you the data you need, and this would be nice and pluggable, not really touching existing code, just enable the optimizer.

If you wanted to log the success, however, I expect you'd need to have a look in MeteringRemoteServiceFactory if that does actually end up loaded.

I dont remember where the whitelist stuff happens if you also wanted to log attempts that were denied

Take it all with a pinch of salt as this all comes only from reading the code, not running it.

So after reading the code-base, asking other people, asking Github Co-Pilot, asking my cat, here is my summarization:

1. Goal: Improve measurement of outgoing federated SERVICE subqueries.
2. Problems:
   • mixed federation and helper
   • not distinguish between successful and failed federations
   • not log outgoing requests and results
3. Potential place to look at (disclaimer: I suspected that these methods were either already implemented or we can implement it ourselves)
   • MeteringRemoteServiceFactory: Extend RemoteServiceCallImpl(params)
     • Timestamp: use method currenttime()
     • Endpoint: use method getServiceURI()
     • Result: try/catch or exception
     • Graph: May be available from the query context or params
   • ServiceRegistry:
     • maybe IsWhitelisted() or similar method exists, I'm not so sure. If we can get the endpoint, we can check the endpoint against Allowlist

Note: Maybe not all data is accessible in this layer, if not we need to investigate how to passing it down from higher layers

Pseudo Java code (check line 30, MeteringRemoteServiceFactory.java) to the best of my knowledge:

[Class, import here]
...
public ICloseableIterator<BindingSet> call(final BindingSet[] bindingSets)
    throws Exception {
    long timestamp = System.currentTime();
    String endpoint = getEndpointUrl(); // method to extract endpoint
    boolean isAllowlisted = checkAllowlist(endpoint);
    String graph = getGraphContext(); // may need to pass this in
    String resultCode = "success";
    try (Timer.Context context = requestTimer.time()) {
        ICloseableIterator<BindingSet> result = super.call(bindingSets);
        // Optionally check result for errors
        logFederationEvent(graph, timestamp, endpoint, isAllowlisted, resultCode);
        return result;
    } catch (Exception e) {
        resultCode = "fail";
        logFederationEvent(graph, timestamp, endpoint, isAllowlisted, resultCode);
        throw e;
    }
}

This seems to remain one of our top priorities for the quarter, so I lifted the deadline from this investigation.

Hi,
We already log the sparql queries in the datalake with this schema.
The missing bits in this schema are:

• the list of federation requests
  • the endpoint hit
  • if this endpoint is allowed or not
  • the status code of the request

Can this information be inferred from the sparql query itself?

• the endpoint hit? yes but based on your experience this is not entirely trivial and requires some effort to parse the sparql query properly
• if this endpoint allowed or not? no, but could be fetched externally once T400807 is done
• the status code of the request? not directly, the sparql query event contains the status code of the host query execution but the queries failing because of the federated endpoints are a subset of all the failed queries

The solutions suggested in this ticket assume that we might be able to get more precise metrics with less engineering efforts with some instrumentation very close to blazegraph internals.
This might not be true:

• As you know blazegraph is no longer maintained and a replacement has to be found, we have strong reservations about adding any new features that depend on blazegraph internals
• A single SERVICE token can result in multiple (sometimes hundreds or even thousands) calls to the federated endpoint which will make it hard to put these counts in perspective of their actual use in user queries.
• The allowlist is evaluated early: RemoteServiceFactory & IASTOptimizers won't see requests to endpoints that are not allowed
• IASTOptimizers are run too early to know if the federated endpoint failed or succeeded

Given the above I believe that the strategy to use blazegraph internals should be avoided.

What are the options:
I believe that the engineering efforts you would have put in this strategy could be allocated to improve the parsing of the sparql query in an analytic context (write a python function usable as a UDF by analysts), I believe that you could get something precise enough for analysis purposes.
The difficult bit is knowing whether the federated endpoint succeeded or failed, this is particularly challenging given that there's no one-to-one mapping between the user-query and the actual HTTP calls to the endpoint.
I think this info could be approximated, esp. if the main product question behind this is knowing what are the federated endpoints we allow that are no longer working: if all the sparql queries using this endpoint fails it might be a good indication that this endpoint is broken.

If this is not enough. You could possibly explore the data available in the webproxy logs, e.g. https://logstash.wikimedia.org/goto/855baabb804e3dbc17e42d65917d1aa4 (I suppose you could create a dedicated dashboard in logstash for this)
We may be OK to add instrumentation to the HTTP client used by blazegraph (as suggested in T391383#10759751). I haven't investigated the approach and have no clue if there are easy listeners you could plug into to add some instrumentation code. This approach implies a new event stream withh all external requests made from blazegraph nodes to the outside world. This dataset will probably be very similar to what's visible in the webproxy logs in logstash so I wonder if this worth the effort.

Caveats:

• the webproxy logs or a new dataset emitted from the HTTP client will be separate from the sparql query logs and might be prone to ambiguous analysis (no one-to-one mapping)
• you'll get all requests made to wikimedia websites via mw:api too
• you'll still need to use the sparql query logs for other analysis (are the endpoints used in queries allowed or not? how many times a federated endpoint is used in user queries?).

Hope this helps, please let us know if you have more questions.

Disclaimer: This is based on my understanding of the codebase and how Blazegraph work. Just one week investigation with completely new knowledge base cannot guarantee that what I understand is completely actionable or correct. Feel free to criticize, modify and adapt.

I combined all the information I've gathered in the last 2 weeks and this outlines the investigation into whether it is possible (or worth it?) to get data on the actual outgoing subqueries:

PROBLEMS:
The current SPARQL query logging has gaps in federation analysis:

• SERVICE endpoints are logged with prefixes instead of expanded IRIs
• Cannot distinguish true federation from helper service invocation
• Cannot tell if federated calls actually succeed/fail
• No mapping between user queries and actual HTTP requests to remote endpoints

SPARQL SERVICE Federation Example:

SELECT ?item ?itemLabel WHERE {
  SERVICE <https://testpedia.org/sparql> {
    ?item wdt:P31 wd:Q146.
    ?item rdfs:label ?itemLabel.
    FILTER(LANG(?itemLabel) = "en")
  }
}

if this query runs on wikidata, we should track:

• Graph (source): wikidata_main
• Target endpoints: https://testpedia.org/sparql
• timestamp
• Success/Failure of the SERVICE call
• Allowlist status of testpedia.org

MISSING METRICS ANALYSIS:
Can we extract these metrics from the SPARQL query itself?

• The target endpoint hit: Yes, but requires effort to parse the SPARQL query properly and extract SERVICE clause
• endpoint Allowlist status: No, but could be fetched externally once T400807 (Expose Allowlist from Wikidata Query Service via API) is done
• The status code of the federation request: Not directly. The sparql query event contains the status code of the host query execution, but queries failing because of federated endpoints are a subset of all failed queries

NOTES:

• Blazegraph is no longer maintained and a replacement must be found -> we need a solution that is not tied to Blazegraph codebase
• The allowlist is evaluated early: RemoteServiceFactory & IASTOptimizers won't see requests to endpoints that are not allowed
• IASTOptimizers run too early to know if federated endpoints failed or succeeded

Taking a closer look into David's recommendation, I believe we can have 3 options:

OPTION 1: Improve SPARQL Query Parsing

• Create Python UDF that extract SERVICE endpoints from query text
• Handle perfixed and full IRIs
• Once T400807 is done, we can cross-reference the endpoint with Allowlist API

OPTION 2: Webproxy Logs

• Logstash dashboard
• Filter federation traffic from MW API calls

OPTION 3: HTTP Proxy/Gateway Instrumentation

• Create a dedicated logging service between Blazegraph and endpoints to log HTTP traffic between them
• This sounds like a "listener" on the infrastructure side
• I suspect it may duplicate webproxy logs(?)

CONCLUSION:

• T400807 is a great starting point.
• Start with improving SPARQL Query Parsing option. Use webproxy logs as validation or fallback solution.

Here is a drafted plan for the implementation:

Here’s a focused action plan for Option 1 (Enhanced SPARQL Query Parsing).

Scope and design
Confirm questions to answer: allowlist compliance, endpoint usage counts, endpoint health.
Finalize enriched schema: source_graph, query_id, event_ts, dt, service_clauses [{endpoint_raw, endpoint_expanded, service_type}], is_allowed, allowlist_rule, parse_errors,... (here are just my suggestions)
Define success metrics and data quality checks.

Parser/UDF
Choose parser approach (Python UDF).
Collect all SERVICE clauses (incl. nested).
Expand prefixes to full IRIs (merge query PREFIX declarations with a default map).
Return structured output; capture parse errors with context.
Unit tests from real query samples and edge cases.

Allowlist integration
Client to fetch/refresh allowlist (supports wildcards).
Implement fast in-memory cache with periodic refresh.
Compute is_allowed and allowlist_rule for each endpoint.

Enrichment job
Read recent sparql/query logs.
Apply parser UDF + allowlist lookup.
Write to partitioned table by dt (keep full event_ts).
Add retries, metrics, and DQ checks (coverage %, error rate, prefix expansion rate).

Populate the new enriched dataset for past history and validation
Pick the most recent 30–90 days.
Validate with spot checks against webproxy/gateway logs (by endpoint + time window).
Verify counts and endpoint detection; separate mwapi traffic when comparing.

Reporting and alerts
Endpoint usage by day and source_graph.
Allowlist compliance (blocked attempts, top offenders).
Heuristic health (failure rates for queries that include an endpoint).
Alerts with thresholds (min volume + failure rate).

I will summarize my understanding of the comments above. Please if you notice that I'm wrong, feel free to correct me.

Context
The goal of this ticket is to investigate the effort required to be able to measure success rate of user attempts to federate out of Wikidata & reasons for failures.
When we have a similar data infrastructure for Cloud, the same solution will be applied to measure federation out of Wikibase Cloud instances.

Measuring use of SERVICE syntax vs. measuring outgoing sub-queries

From what I understood from @dcausse 's explanation, a single use of SERVICE syntax may produce numerous HTTP requests, so measuring outgoing requests is not ideal (unless we could deduplicate such subqueries). We also should avoid doing any changes in Blazegraph, as it will be replaced by another solution.
The alternative solution proposed in Option 1 is to improve the parsing logic to solve the problems that currently exist there. This includes:

• Resolving prefixes to full URLs T399805
• Recognizing federation from invoking helpers T399827
• Recognizing which use of SERVICE syntax resulted in a successful or failed federation and for what reason T399828

Btw, this ticket was actually created as a workaround to Option 1, to avoid solving all these problems on the parser level. We initially assumed it would be easier to look at actual calls happening, rather than what users wrote, especially because we want to understand the status of the subquery.

Considerations around measuring the use of SERVICE syntax

If we're exploring Option 1, here are some things to be mindful of:

1. Except for the prefixes that users explicitly wrote in the query, we should resolve native and configured prefixes. @dcausse pointed me to the WDQS config and code and configuration in which you start WDQS. Apparenty, prefixes for wd: and wdt: should be part of the latter one, but I couldn't personally figure out the exact place where these are configured. This might be wdt:, but where is wd: ? (It might also be important to know in the future for us, because we want to keep wd: for Wikidata and allow other Wikibases specify their own preferred default prefix like wb: - T335448)

2. We still need to find a way to differentiate between federation attempts to allowlisted endpoints, federation attempts to non-allowlisted endpoints (operational or not) and invoking behavior of helper services like wikibase:label. It's still unclear to me at this point how this could be done without looking whether the use of SERVICE syntax produces a sub-query. @dcausse Could you suggest some approach here?

3. We won't be able to see the status of subquery, only status of the original query. We will need to guess from the status whether the failure is related to one of federated endpoints not being on the Allowlist, one of federated endpoints not operating, or other reasons. If an original query is trying to federate to 2+ endpoints, the results might be confusing.

Adjusted expectations
For this reason, it seems like the expected structure of the data should be stored on the level of the original executed user query (rather than each sub-query it produced) and provide the following information:

• graph name (In this context, “graph” refers to the source database/instance where the user originally ran their query. e.g. wikidata_main, wikidata_scholarly,.. etc)
• timestamp
• result code (success / fail)
• if fail, some kind of error code or description if it's available
• array of endpoints this query federated to; for each endpoint
  • URI
  • is it on the allowlist
  • is it known to be operational (example of this check is done here, could be done daily, for example)