This query is returning truncated results for me:
If I download it from the browser, the results are complete and the file size is 76.3 MB. However, if I download it through Curl or Python, I receive an unreadable XML file which has been truncated at 3.6 MB (same with JSON). Any idea why this might be?
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| cache_misc: remove all do_stream=true | operations/puppet | production | +0 -14 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | BBlack | T133490 Wikidata Query Service REST endpoint returns truncated results | |||
| Resolved | • ema | T131501 Convert misc cluster to Varnish 4 | |||
| Resolved | • ema | T134989 WDQS empty response - transfer clsoed with 15042 bytes remaining to read |
Here is the SPARQL, for clarity:
SELECT ?qid ?imdb WHERE {?qid wdt:P345 ?imdb. FILTER regex(str(?imdb), "nm|tt")} ORDER BY ?qidA similar issue has been reported by Ivan A. Krestinin here: https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team#Corrupted_response_end.
It happens to him with Wikidata API requests, not WQS, but it has oddly similar features:
I have noticed that if I set the --compressed option in Curl the file does not get truncated. I searched on Gerrit and found a Varnish configuration change from April 11 whose commit message says:
Note we still need to re-test assumptions about do_stream=false adding Content-Length to responses which lacked them in the common case (no gzip/gunzip on the fly on the response side).
Commit here: https://gerrit.wikimedia.org/r/#/c/282716. Related tasks: T128813, T131501, T131761
We now have some understanding of the mechanism of this bug ( T133866#2275985 ). It should go away in the imminent varnish 4 upgrade of the misc cluster in T131501.
Change 287633 had a related patch set uploaded (by BBlack):
cache_misc: remove all do_stream=true
This works now. There's a significant pause at the start of the transfer from the user's perspective if it's not a cache hit, because streaming is disabled as a workaround (so it has to completely load the data into each cache layer before starting the data stream to the user), but it does function correctly. The non-streamed pause behavior will go away with T131501 .