I think there are two ways to achieve this.

New endpoint from wdqs

• Generalize org.wikidata.query.rdf.blazegraph.WikibaseContextListener#loadAllowlist as a FederatedEndpointsLoader so that it loads a FederatedEndpoints POJO representing the endpoints instead of directly registering them to the ServiceRegistry.
• Adapt WikibaseContextListener#loadAllowlist to use this new FederatedEndpointsLoader and register the endpoints listed in FederatedEndpoints
• Create a new servlet in the blazegraph submodule that uses the FederatedEndpointsLoader and serializes it to the format we want. Creating a new servlet requires:
  • A new servlet class (see LDFServlet)
  • Declaring the servlet in src/main/webapp/WEB-INF/web.xml
  • (WMF specific) Adapt modules/query_service/templates/nginx.erb to forward a new location to this servlet.

Pros:

• Allows to include some logic to manipulate the list at runtime

Cons:

• Blazegraph specific, might have to be re-implemented when we replace blazegraph

Static resource generated by puppet and served by nginx

• Introduce a new hiera value used by all query_service nodes that contains all external federated endpoints
• Pass this list to the modules/query_service/templates/allowlist.txt.epp template
• Introduce a new template that generates the response we want from this endpoint to a location available by nginx
• Add a new location to modules/query_service/templates/nginx.erb to return this static file

Pros:

• blazegraph agnostic

Cons:

• limited to what's doable via puppet template
• if this endpoint needs to be exposed from wikibase cloud sparql servers this might require the same logic to be implemented in its infra

At first glance the latter sounds a lot easier to me.

@sowmya.guru to discuss with Anton the priority and how much effort should be spent on an automatic solution

In T400807#11084795, @dcausse wrote:

At first glance the former sounds a lot easier to me.

So I meant the latter here, working at the puppet/nginx level will certainly require less effort.

Thank you @dcausse. Appreciate your input on this! :)

With a slightly more wikibase ecosystem hat on

New endpoint from wdqs

Pros:

• Will be included in wdqs deployments for all wikibase / wdqs users, in a consistent way with little other configuration and setup (when they adopt the version it is released in)

Cons:

• Indeed, if the backend changes, this would need to be reworked

Static resources

Pros:

• Cloud does actually have 2 levels of nginx, and one other service that this sort of thing could be added to
• wikibase "suite" / "deploy" has traefik that this could be added to https://github.com/wmde/wikibase-release-pipeline/blob/main/deploy/config/traefik-dynamic.yml

Cons:

• This only solves the issue for wikidata.org? if this is the whitelist from WMF puppet that is being used.
• Not all wikibase deployments have nginx in the say way in front of their wdqs deployment

Just my 5 cents: the scope of this specific ticket is to start with Wikidata. But, of course, we are looking for a solution that could eventually be applied to other Wikibases in the ecosystem. So we will have to make trade offs :)

In T400807#11093115, @Anton.Kokh wrote:

Just my 5 cents: the scope of this specific ticket is to start with Wikidata. But, of course, we are looking for a solution that could eventually be applied to other Wikibases in the ecosystem. So we will have to make trade offs :)

With that context, perhaps the static file ends up being easier!

The confusion likely comes from the fact that:

• we still call the query service for wikibase both wdqs and Wikidata Query Service. See T321393: Consider renaming the 'Wikidata Query Service' software to 'Wikibase Query Service'
• This ticket is only tagged with Wikibase Cloud, perhaps it needs Wikidata and Wikidata-Query-Service too?

@dcausse I will copy-paste the information you shared with me via email, so it's visible to others. I also updated the body of the ticket to explicitly call internal federation points out.

While implementing this endpoint in blazegraph you'll probably stumble on internal federation and you can make the decision to expose those or not. In other words you'll have the choice to list all or some of those.
I guess it depends on why you want this list as a client of this endpoint.

1/ I want to see the federated endpoints so that I can replicate them in my own query service
For this you want the endpoint to be a valid public endpoints, this might be false for internal endpoints:

• https://wdqs-scholarly.discovery.wmnet/sparql should clearly not be used outside the WMF infrastructure
• https://query.wikidata.org/subgraph/scholarly_articles is here mainly to allow syntactic sugar and allow writing SERVICE wdsubgraph:scholarly_articles
• https://query-scholarly.wikidata.org/sparql is valid even outside the WMF infra

2/ You want to inform an analisys that parses the WMF SPARQL query logs and answer a question like "How many queries are trying to federate against an endpoint that is not allowed"
For this you can use the internal federated endpoints.

In our case, we want the API to potentially serve both purposes you mentioned and even 1 additional one.

1. To allow query federation from Cloud instances to the same endpoints that are allowed on Wikidata graphs (T400806)
2. To let users know which endpoints they can federate to directly in QS UI (T381641)
3. To potentially automate analysis of the query logs and see how many queries attempted federation with non-allowlisted endpoints (T399829)

You mentioned that https://wdqs-scholarly.discovery.wmnet/sparql (and it's -main version) should clearly not be used outside the WMF infrastructure, so they should not show up in (1) and (2), but should probably show up in (3).
Do you see any problems with such solution where the internal federated endpoint and its aliases are separated?

As I understood it actually implementing this is now maybe in the hands of the Wikidata Platform Team and the motivation / nudging would move to Wikibase Reuse Team

I think the Allowlist goals/KRs are still owned by Cloud/Anton so I'd leave it with him unless that is not the case

@Ifrahkhanyaree_WMDE Our goals and KRs currently relate to growing the Allowlist and making sure the endpoints on it are operational. This task was an enabler for some things we were planning to 'improve query federation' (the objective that shifted your way). Depending on what KRs and plans you have there, you may or may not want this to be implemented. I would say this task is now in limbo until further notice, but for sure will need to be tackled by the WMF Wikidata Platform team if someone ever needs it again in the context of their work.

Given the move away from Blazegraph, we imagine the future implementation of this needs to be taken as part of the requirements for the new backend.
E.g. it might be part of the solution for this ticket T406658: Create automation to verify WDQS allowlist operations

Thanks, Anton. We'll reflect in our planning.