Page MenuHomePhabricator

wdqs-qlever: federation and eventgate posting don't work at all on k8s due to network policy
Closed, DeclinedPublic

Description

This is sort of a companion to T431976 in that it involves getting federation working, but it's actually about getting egress to work at all vs ensuring that federating between main and scholarly works (which also involves possible internal routing and user-agent allowlists).

Here's the more general problem. From a stat host, I tried

lerickson@stat1008:~$ curl -X POST https://query-next.wikidata.org/sparql -H "Content-Type: application/sparql-query" -d 'select * {
  SERVICE <http://rdf.insee.fr/sparql> {
    SELECT * {
      ?s ?p ?o .
    } LIMIT 1
  }
}'

This returns a not very descriptive 500 - Internal Server Error. In the qlever logs I can see the query was received and federation was attempted, but there is nothing after that. In the proxy logs I see: "message":"The timeout period of 30000ms has been exceeded while executing POST /sparql/sparql for server wdqs-main-external-qlever-service:7001" I think the "/sparql/sparql" thing in there looks weird but I don't believe it actually points to a problem, because the proxy can interact with the backend just fine, so I am not worrying about that. But the timeout is interesting and suggests that outbound requests are failing. (BTW, I did try this query on a non-k8s setup locally and it worked, so it should return a result.)

Indeed, when I run kubectl describe netpol wdqs-main-external -n wdqs I see this:

Spec:
  PodSelector:     app=wdqs,release=main-external
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From: <any> (traffic not restricted by source)
  Allowing egress traffic:
    <none> (Selected pods are isolated for egress connectivity)
  Policy Types: Egress, Ingress

which makes me think all egress is blocked.

Additional evidence comes from the proxy logs about eventgate posting, which say thing like

{"timestamp":"2026-07-24T20:48:05.387500268Z","sequence":441,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.wikimedia.wdqs.events.QueryEventFilters","level":"WARN","message":"Event posting failure: connection timed out after 15000 ms: eventgate-analytics.discovery.wmnet/10.2.2.42:4592","threadName":"vert.x-eventloop-thread-1","threadId":25,"mdc":{},"ndc":"","hostName":"wdqs-main-external-proxy-79b7f66d87-7cbc2","processName":"/usr/lib/jvm/java-25-openjdk-amd64/bin/java","processId":1}

Eventgate posting also works fine on my local non-k8s setup.

Event Timeline

Update: I was wrong above when I concluded that no egress is possible. Our k8s services can currently reach gitlab (global egress config here) and also s3 and kafka (dc-specific egress config here). This is just not reflected in the network policies I found.

It's still a problem that we can't egress to all these targets. The solutions, though, will probably be different:

  1. Internal query federation between main and scholarly: already targeted in T431976, just mentioning here for completeness
  2. Eventgate: created T433375
  3. External federation: for the external deployment, I think this will work after https://github.com/ad-freiburg/qlever/issues/3129 is resolved. TBD about the future internal deployment.

Federation works. Eventgate still doesn't but it's being tracked in that other ticket above. I think we can close this now since the remaining work is small and covered by a task.