Page MenuHomePhabricator
Create Task
Maniphest T225523

U+0200F in query service response for GND ID
Closed, ResolvedPublicBUG REPORT
Actions

Description

Problem:
About 60 GND ID statements have an invisible UTF8 markers at the end of the value when accessed via the query service. It's U+0200F (“RIGHT-TO-LEFT MARK”, https://decodeunicode.org/en/u+0200F).
These values have been entered by different users.
The values are fine in the Item UI as well as the JSON and TTL exports. They show the issue in the csv export and query service UI.

Example:
This file contains Items that have a GND ID statement with the issue as well as the values for these statments:

vollständigeListe_WD-GNDID_mit200f.txt3 KBDownload

Possibly related:

Details

SubjectRepoBranchLines +/-
Filter invisible characters that cause trouble in Blazegraph ICU collationwikidata/query/rdfmaster+73 -7
Customize query in gerrit

Related Objects

Event Timeline

Lydia_Pintscher created this task.Jun 11 2019, 3:10 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptJun 11 2019, 3:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Lydia_Pintscher updated the task description. (Show Details)Jun 11 2019, 3:12 PM
elal subscribed.Jun 11 2019, 3:14 PM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)Jun 11 2019, 3:31 PM
Smalyshev changed the subtype of this task from "Task" to "Bug Report".Jun 11 2019, 9:42 PM
Smalyshev added subscribers: Igorkim78, Smalyshev.

It's the same issue as T197447. We probably need to think about more systemic solution for this... Changing ICU collation may increase data size and lead to other complications, so maybe filtering the data and removing characters like U+0200F?

Smalyshev triaged this task as Medium priority.Jun 12 2019, 4:57 AM
Smalyshev added a project: Discovery-Wikidata-Query-Service-Sprint.
Smalyshev moved this task from Backlog to In progress on the Discovery-Wikidata-Query-Service-Sprint board.Jun 13 2019, 6:50 AM
gerritbot added a comment.Jun 14 2019, 12:47 AM

Change 516981 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] Filter invisible characters that cause trouble in Blazegraph ICU collation

https://gerrit.wikimedia.org/r/516981

gerritbot added a project: Patch-For-Review.Jun 14 2019, 12:47 AM
gerritbot added a comment.Jun 14 2019, 5:07 PM

Change 516981 merged by jenkins-bot:
[wikidata/query/rdf@master] Filter invisible characters that cause trouble in Blazegraph ICU collation

https://gerrit.wikimedia.org/r/516981

Maintenance_bot removed a project: Patch-For-Review.Jun 14 2019, 5:10 PM
Smalyshev moved this task from In progress to Done on the Discovery-Wikidata-Query-Service-Sprint board.Jun 17 2019, 7:00 AM
Smalyshev closed this task as Resolved.Jun 17 2019, 9:57 PM
Smalyshev claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL