Page MenuHomePhabricator

Query service throws exception for non-English wikis
Closed, DuplicatePublic

Description

The query service fails when it gets updates from a non-English wiki, presumably since it is assuming "Special:" and breaks when it gets an internationalized version like "Spezial".

(Probably relevant code as identified by @Addshore: https://github.com/wikimedia/wikidata-query-rdf/blob/d52b50c57b739ca790726b4e72095ac3a06cfbbb/common/src/main/java/org/wikidata/query/rdf/common/uri/DefaultUrisScheme.java#L76-L77)

dnb_wdqs-updater   | 22:48:12.839 [main] INFO  o.w.q.r.t.change.RecentChangesPoller - Got 1 changes, from Q1@10@20210310224811|10 to Q1@10@20210310224811|10
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized subjects: [https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1] while processing https://test42.wikibase.wiki/entity/Q1.  Expected only sitelinks and subjects starting with https://test42.wikibase.wiki/wiki/Special:EntityData/ and [https://test42.wikibase.wiki/entity/]
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1 p:http://www.w3.org/1999/02/22-rdf-syntax-ns#type o:http://schema.org/Dataset
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1 p:http://schema.org/about o:https://test42.wikibase.wiki/entity/Q1
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1 p:http://creativecommons.org/ns#license o:http://creativecommons.org/publicdomain/zero/1.0/
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1 p:http://schema.org/softwareVersion o:"1.0.0"^^<http://www.w3.org/2001/XMLSchema#string>
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1 p:http://schema.org/version o:"10"^^<http://www.w3.org/2001/XMLSchema#integer>
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1 p:http://schema.org/dateModified o:"2021-03-10T22:48:11Z"^^<http://www.w3.org/2001/XMLSchema#dateTime>
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1 p:http://wikiba.se/ontology#statements o:"2"^^<http://www.w3.org/2001/XMLSchema#integer>
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1 p:http://wikiba.se/ontology#identifiers o:"0"^^<http://www.w3.org/2001/XMLSchema#integer>
dnb_wdqs-updater   | 22:48:18.029 [update 3] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:https://test42.wikibase.wiki/wiki/Spezial:EntityData/Q1 p:http://wikiba.se/ontology#sitelinks o:"0"^^<http://www.w3.org/2001/XMLSchema#integer>
dnb_wdqs-updater   | 22:48:18.030 [update 3] WARN  org.wikidata.query.rdf.tool.Updater - Contained error syncing.  Giving up on Q1
dnb_wdqs-updater   | org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [(https://test42.wikibase.wiki/entity/Q1, http://www.w3.org/2000/01/rdf-schema#label, "MyItem"@en), (https://test42.wikibase.wiki/entity/Q1, http://www.w3.org/2000/01/rdf-schema#label, "GermanLabel"@de), (https://test42.wikibase.wiki/entity/Q1, https://test42.wikibase.wiki/prop/direct/P1, "pewpew"^^<http://www.w3.org/2001/XMLSchema#string>), (https://test42.wikibase.wiki/entity/Q1, https://test42.wikibase.wiki/prop/direct/P1, "hi"^^<http://www.w3.org/2001/XMLSchema#string>), (https://test42.wikibase.wiki/entity/Q1, https://test42.wikibase.wiki/prop/P1, https://test42.wikibase.wiki/entity/statement/Q1-b58a5bb4-48d3-c7df-db01-f6e289c30525), (https://test42.wikibase.wiki/entity/statement/Q1-b58a5bb4-48d3-c7df-db01-f6e289c30525, http://www.w3.org/1999/02/22-rdf-syntax-ns#type, http://wikiba.se/ontology#BestRank), (https://test42.wikibase.wiki/entity/statement/Q1-b58a5bb4-48d3-c7df-db01-f6e289c30525, http://wikiba.se/ontology#rank, http://wikiba.se/ontology#NormalRank), (https://test42.wikibase.wiki/entity/statement/Q1-b58a5bb4-48d3-c7df-db01-f6e289c30525, https://test42.wikibase.wiki/prop/statement/P1, "pewpew"^^<http://www.w3.org/2001/XMLSchema#string>), (https://test42.wikibase.wiki/entity/Q1, https://test42.wikibase.wiki/prop/P1, https://test42.wikibase.wiki/entity/statement/Q1-45d3cd1c-49f0-8ed8-95a6-3ee5bb1b36c7), (https://test42.wikibase.wiki/entity/statement/Q1-45d3cd1c-49f0-8ed8-95a6-3ee5bb1b36c7, http://www.w3.org/1999/02/22-rdf-syntax-ns#type, http://wikiba.se/ontology#BestRank), (https://test42.wikibase.wiki/entity/statement/Q1-45d3cd1c-49f0-8ed8-95a6-3ee5bb1b36c7, http://wikiba.se/ontology#rank, http://wikiba.se/ontology#NormalRank), (https://test42.wikibase.wiki/entity/statement/Q1-45d3cd1c-49f0-8ed8-95a6-3ee5bb1b36c7, https://test42.wikibase.wiki/prop/statement/P1, "hi"^^<http://www.w3.org/2001/XMLSchema#string>)]

The above error is from when using the stock WMDE Wikibase-Docker image: https://github.com/wmde/wikibase-docker/tree/master/wdqs/0.3.40. It does not happen for English language wikis.

This is a problem for the German National Library and for the Kunstmuseum The Hague. And anyone else that wants to run a non-English Wikibase wiki.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I have reproduced the issue by running a Wikibase in both the Japanese and Korean language versions as configured using https://github.com/andrawaag/wikibase_languages

wdqs-updater_1     | 22:28:12.349 [main] INFO  o.w.q.r.t.change.RecentChangesPoller - Got no real changes
wdqs-updater_1     | 22:28:12.349 [main] INFO  org.wikidata.query.rdf.tool.Updater - Sleeping for 10 secs
wdqs-updater_1     | 22:28:22.401 [main] INFO  o.w.q.r.t.change.RecentChangesPoller - Got 1 changes, from Q1@2@20210315222813|2 to Q1@2@20210315222813|2
wdqs-updater_1     | 22:28:23.011 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized subjects: [http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1] while processing http://wikibase.svc/entity/Q1.  Expected only sitelinks and subjects starting with http://wikibase.svc/wiki/Special:EntityData/ and [http://wikibase.svc/entity/]
wdqs-updater_1     | 22:28:23.017 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1 p:http://www.w3.org/1999/02/22-rdf-syntax-ns#type o:http://schema.org/Dataset
wdqs-updater_1     | 22:28:23.018 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1 p:http://schema.org/about o:http://wikibase.svc/entity/Q1
wdqs-updater_1     | 22:28:23.018 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1 p:http://creativecommons.org/ns#license o:http://creativecommons.org/publicdomain/zero/1.0/
wdqs-updater_1     | 22:28:23.018 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1 p:http://schema.org/softwareVersion o:"1.0.0"^^<http://www.w3.org/2001/XMLSchema#string>
wdqs-updater_1     | 22:28:23.019 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1 p:http://schema.org/version o:"2"^^<http://www.w3.org/2001/XMLSchema#integer>
wdqs-updater_1     | 22:28:23.019 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1 p:http://schema.org/dateModified o:"2021-03-15T22:28:13Z"^^<http://www.w3.org/2001/XMLSchema#dateTime>
wdqs-updater_1     | 22:28:23.019 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1 p:http://wikiba.se/ontology#statements o:"0"^^<http://www.w3.org/2001/XMLSchema#integer>
wdqs-updater_1     | 22:28:23.019 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1 p:http://wikiba.se/ontology#identifiers o:"0"^^<http://www.w3.org/2001/XMLSchema#integer>
wdqs-updater_1     | 22:28:23.019 [update 0] INFO  o.wikidata.query.rdf.tool.rdf.Munger - Unrecognized statement: s:http://wikibase.svc/wiki/%ED%8A%B9%EC%88%98:EntityData/Q1 p:http://wikiba.se/ontology#sitelinks o:"0"^^<http://www.w3.org/2001/XMLSchema#integer>
wdqs-updater_1     | 22:28:23.511 [main] INFO  org.wikidata.query.rdf.tool.Updater - Polled up to 2021-03-15T22:28:13Z at (0.0, 0.0, 0.0) updates per second and (0.0, 0.0, 0.0) milliseconds per second
wdqs-updater_1     | 22:28:23.538 [main] INFO  o.w.q.r.t.change.RecentChangesPoller - Got no real changes
wdqs-updater_1     | 22:28:23.538 [main] INFO  org.wikidata.query.rdf.tool.Updater - Sleeping for 10 secs

Steps I followed:

  1. Pulled the above mentioned repository from gihub
  2. Installed docker & Docker-compose
  3. sudo docker-compose up.

This leads to

And no updates on the wbqs.

Tentatively closing as a duplicate of T263427 as this sounds very similar, please re-open if you think it's completely different or if the workaround mentioned there does not work for you.