Page MenuHomePhabricator
Create Task
Maniphest T202785

Federation request to https://ld.stadt-zuerich.ch/query fails
Closed, ResolvedPublic
Actions

Description

Query:

SELECT *
WHERE 
{
  SERVICE <https://ld.stadt-zuerich.ch/query> {
    SELECT * WHERE {
      ?Kennzahl a <http://purl.org/linked-data/cube#AttributeProperty> ;
      <http://www.w3.org/2000/01/rdf-schema#label> ?KennzahlLabel .
    } 
  }
}

Fails with:

Caused by: javax.net.ssl.SSLException: Received fatal alert: protocol_version
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
	at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:507)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:807)
	at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:399)
	at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:97)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
	... 1 more

Related Objects
Search...

Event Timeline

Smalyshev created this task.Aug 24 2018, 11:14 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptAug 24 2018, 11:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev renamed this task from Federation request to fails to Federation request to https://ld.stadt-zuerich.ch/query fails.Aug 24 2018, 11:14 PM
Smalyshev claimed this task.Aug 24 2018, 11:18 PM
Smalyshev triaged this task as Medium priority.
Smalyshev added a subscriber: Gehel.Aug 27 2018, 11:42 PM

I suspect the problem may be in HTTP/2. Looking into communication with ld.stadt-zuerich.ch, it seems to be switching to HTTP/2. https://www.eclipse.org/jetty/documentation/current/http2.html says:

To use HTTP/2 in Jetty via a TLS connector you need to add the ALPN boot jar in the boot classpath. This is done automatically when using the Jetty distribution’s start.jar module system, but must be configured directly otherwise.

Maybe this is missing. @Gehel, I wonder if you know anything about this?

Smalyshev added a comment.EditedAug 28 2018, 12:23 AM

From this: https://stackoverflow.com/questions/47730864/jetty-client-cant-connect-to-jetty-server-via-http-2 it sounds like there's no OOB support in Jetty for going from HTTP/1 to HTTP/2, at least not without some coding. So I am not sure how to support this now. Specifically:

that is correct, there is no way to do it currently. An instance of HttpClient is created with a specific transport and currently cannot switch to another transport.

You can partially workaround this by having two HttpClient instances, one with the HTTP/1.1 transport and one with the HTTP/2 transport, but you have to try one, if it fails try the other, it's clunky.

Since there's no way for us to know which transport each URL is using, and Blazegraph uses HTTP/1 transport, I assume that means we can't support HTTP/2 servers for federation at least until Jetty supports this.

Mchlrch subscribed.EditedAug 28 2018, 12:17 PM

The endpoint https://ld.stadt-zuerich.ch/query supports HTTP/1.0, HTTP/1.1 and HTTP/2. I defaults to HTTP/2, if the client does not request an explicit version.

@Smalyshev How about explicitly specifying HTTP/1.1 for the jetty-client?

Query:

$ cat foo.rq 
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT * WHERE {
  GRAPH <https://linked.opendata.swiss/graph/zh/statistics> {
    ?sub ?pred ?obj .
  } 
}
LIMIT 1

curl for HTTP/1.0:

$ curl --http1.0 -i --data-urlencode query@foo.rq https://ld.stadt-zuerich.ch/query
HTTP/1.0 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public, max-age=120
Content-Type: application/sparql-results+xml
Date: Tue, 28 Aug 2018 11:54:40 GMT
Set-Cookie: _f3bc9=http://10.42.34.100:80; Path=/
Vary: Accept-Encoding
Vary: Accept, Accept-Encoding
Via: 1.1 varnish-v4
X-Varnish: 296846
Content-Length: 540

<?xml version='1.0' encoding='UTF-8'?>
<sparql xmlns='http://www.w3.org/2005/sparql-results#'>
	<head>
		<variable name='sub'/>
		<variable name='pred'/>
		<variable name='obj'/>
	</head>
	<results>
		<result>
			<binding name='sub'>
				<uri>https://ld.stadt-zuerich.ch/.well-known/void</uri>
			</binding>
			<binding name='pred'>
				<uri>http://www.w3.org/1999/02/22-rdf-syntax-ns#type</uri>
			</binding>
			<binding name='obj'>
				<uri>http://rdfs.org/ns/void#DatasetDescription</uri>
			</binding>
		</result>
	</results>
</sparql>

curl for HTTP/1.1:

$ curl --http1.1 -i --data-urlencode query@foo.rq https://ld.stadt-zuerich.ch/query
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public, max-age=120
Content-Type: application/sparql-results+xml
Date: Tue, 28 Aug 2018 11:54:18 GMT
Set-Cookie: _f3bc9=http://10.42.34.100:80; Path=/
Vary: Accept-Encoding
Vary: Accept, Accept-Encoding
Via: 1.1 varnish-v4
X-Varnish: 1540376
Content-Length: 540

<?xml version='1.0' encoding='UTF-8'?>
<sparql xmlns='http://www.w3.org/2005/sparql-results#'>
	<head>
		<variable name='sub'/>
		<variable name='pred'/>
		<variable name='obj'/>
	</head>
	<results>
		<result>
			<binding name='sub'>
				<uri>https://ld.stadt-zuerich.ch/.well-known/void</uri>
			</binding>
			<binding name='pred'>
				<uri>http://www.w3.org/1999/02/22-rdf-syntax-ns#type</uri>
			</binding>
			<binding name='obj'>
				<uri>http://rdfs.org/ns/void#DatasetDescription</uri>
			</binding>
		</result>
	</results>
</sparql>
Mchlrch added a comment.Aug 28 2018, 1:01 PM

Allow me to reformulate: How about making the jetty-client ask explicitly for HTTP/1.1?

Smalyshev added a comment.Aug 28 2018, 10:35 PM

The problem is that asking explicitly seems to be what ALPN is doing, and I am not sure Jetty version we're using supports that. Additionally, ALPN jar versions appear to be linked specifically to single JVM version, which means the binary would run only on single JVM - that's not a very good idea. If there's any other way of asking the server to keep HTTP/1.1 in Jetty, I'm open to suggestions.

Smalyshev moved this task from Incoming to Current work on the Wikidata-Query-Service board.Aug 28 2018, 11:14 PM
Mchlrch added a comment.Aug 29 2018, 12:07 AM

I understand your concerns that bringing in ALPN would most likely cause problems, if it even would be possible at all. I suggest to explore if we can find a resolution without bringing in new dependencies.

I suggest the following procedure:

  1. isolate the code that creates the HttpClient in order to make the issue reproducible outside the Query-Service codebase, in order to confirm that we are really searching in the right spot
  2. explore if the setup of the HttpClient could be modified in order to make the request work
  3. in case of success, apply the necessary changes to the Query-Service codebase

I could help with step 1) and 2).

I cloned WDQR [1] and had a look at the source, but I'm not even sure, that this is the right repository to look at. If it is, then it looks like the HttpClient is created in either of the following two places:

  • module blazegraph: org.wikidata.query.rdf.blazegraph.ProxiedHttpConnectionFactory, line 40
  • module tools: org.wikidata.query.rdf.tool.HttpClientUtils, line 77

Do you know which of the two places is the right one to investigate? Or could you give me a pointer to the right source, in case I'm searching in the wrong spot.

According to the parent POM, the Query-Service runs with Java 8, right?

Smalyshev added a comment.Aug 29 2018, 5:25 AM

ProxiedHttpConnectionFactory is the one used for federated connections (it doesn't do much except invoking DefaultHttpClientFactory() and assigning proxy, the other one is used for the Updater and is irrelevant for current case.

I do not see where there's a place for specifying protocol version. I think it's part of ALPN which is a separate extension and not very well supported in Java 8.

I would expect in the absence of such support the server would default to HTTP/1.1, as HTTP/2 client surely can negotiate the connection correctly, but HTTP/1.1 can't.

According to the parent POM, the Query-Service runs with Java 8, right?

Yes, it runs on Java 8.

Mchlrch added a comment.Aug 29 2018, 10:28 PM

I was able to reproduce the issue outside the WDQR codebase [1], using the blazegraph bigdata-client library.

Unfortunately, I can only confirm what Stas already suspected and commented earlier: There is no easy fix.

What I conclude, is that current WDQR does not support federation to endpoints that also provide HTTP/2 over TLS, because HTTP/2 over TLS requires the use of ALPN. The blazegraph bigdata-client library used in WDQR uses jetty version 9.2.3.v20140905, which predates HTTP/2 standardization from 2015 and therefore lacks ALPN support.

WDQR federation however does work with endpoints that also provide HTTP/2 but without TLS - because negotation works without ALPN in that case.

Looking forward, it would seem reasonable to aim at including support for HTTP/2 over TLS in WDQR. Jetty apparently supports HTTP/2 starting with version 9.3 [2], so maybe upgrading blazegraph to jetty 9.3 is doable. On the other hand, I noticed that the blazegraph codebase has not seen any activity in the last two years, which might also be a motivation for Wikidata to move on to another datastore.

To resolve on the endpoint side would apparently require to completely drop support for HTTP/2 over TLS. Even though theoretically possible, I personally don't consider that an option, because it's just too regressive.

Probably just a bad idea, but I mention it anyway: Another option (pretty far fetched and adventurous) might be to have a MITMing proxy inside Wikidata infrastructure, that terminates outbound HTTP/2-TLS on one side and exposes HTTP/1.1-TLS to WDQR. Also I have no clue if there are products, that actually support such behavior.

1: https://github.com/mchlrch/wdqr-ssz-tc/tree/master/wdqr-ssz-tc
2: http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/VERSION.txt

Smalyshev added a comment.Aug 29 2018, 10:54 PM

We do use proxy anyway but I don't think it's doing any fancy HTTP 2<->1 stuff. The proxy server is Squid, I'll try to find out if it can do this.

Addshore moved this task from incoming to monitoring on the Wikidata board.Aug 30 2018, 9:07 AM
Gehel added a comment.Aug 31 2018, 7:58 PM

Looking a bit into this, it does not look like blazegraph has a deep integration with Jetty (why does it even have any dependency on Jetty is a mystery to me). So repackaging with a more recent jetty-http (or the whole jetty stack) might not be that hard (well, it is trivial to upgrade, non trivial to test).

For the record, OkHttp is a pretty nice HTTP client that supports HTTP 2 out of the box on Java 8 (but we're not going to switch HTTP clients inside of Blazegraph, that sounds too crazy).

Smalyshev removed Smalyshev as the assignee of this task.Sep 10 2018, 11:45 PM

So repackaging with a more recent jetty-http (or the whole jetty stack) might not be that hard

But I understand that the proper support for the scenario we need is only available with Java 9?

Smalyshev changed the task status from Open to Stalled.Oct 12 2018, 8:56 PM
Smalyshev moved this task from Current work to Incoming on the Wikidata-Query-Service board.
Smalyshev added a parent task: T197530: [tracking] federation query issues on Wikidata Query Server.Oct 14 2018, 7:15 PM
Smalyshev mentioned this in T195203: Federated SPARQL queries https://data.pdok.nl/sparql failing with error 500.Oct 14 2018, 9:55 PM
Smalyshev closed this task as Resolved.Nov 9 2018, 9:14 PM
Smalyshev claimed this task.

Seems to be OK now.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits