When running the harvester in docker for the nl-wd_(nl) dataset (in the Wikidata branch) I get
HTTPSConnectionPool(host='query.wikidata.org', port=443): Read timed out. (read timeout=30)
Repeating the run makes it work so I'm guessing the sparql query finishes correctly but the harvester is simply not patient enough.
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T138668 Harvest Wikidata into the Monuments database | |||
| Resolved | Lokal_Profil | T172772 Wlm/heritage/erfgoedbot sprint at Wikimania Hackathon 2017 | |||
| Resolved | Lokal_Profil | T172690 Investigate time-out issues with Wikidata harvests |
Event Timeline
Comment Actions
So a quick and dirty solution might be to catch the timeout. Sleep for a short time then re-run. That way the query will finish on the sparql end and we get the result.
The sleep would have to be short enough that the sparql endpoint doesn't decide to invalidate the data.
Comment Actions
Change 371602 had a related patch set uploaded (by Lokal Profil; owner: Lokal Profil):
[labs/tools/heritage@wikidata] Handle sparql http timeout issue
Comment Actions
Change 371602 merged by jenkins-bot:
[labs/tools/heritage@wikidata] Handle sparql http timeout issue
Comment Actions
Additionally an upstream fix in pywikibot has been made in
https://gerrit.wikimedia.org/r/371697
Comment Actions
I am not sure the timeout catch actually works: testing the nl_nl, I got the timeout on the Query service UI, but on pywikibot side I just had an empty result.
Could something have changed with the query service timeout reporting?
Comment Actions
Change 532414 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Improve SPARQL efficiency for nl_nl
Comment Actions
Change 532414 merged by jenkins-bot:
[labs/tools/heritage@master] Improve SPARQL efficiency for nl_nl and au_en
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2019-08-29T20:49:21Z] <JeanFred> Deploy latest from Git master: c518c02 (T172690)
Comment Actions
@Lokal_Profil / @JeanFred: All related patches in Gerrit have been merged, so I'm boldly resolving this task.
(Please reopen if there is more to do in this task - thanks!)