Task Breakdown Notes:

• This might be as straight forward as using the built in JS URL decoder to decode the url
• In order to protect against malicious queries (these should never be in real sitelink URLs), don’t decode (or, re-encode after decoding) stuff like
  • whitespace characters
  • control characters
  • bidirectional overrides
  • etc. (could be done depending on the Unicode category – we probably want to unescape letters, unescape punctuation, but everything else is TBD)

Change 888052 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[wikidata/query/gui@master] Decode URIs for display

https://gerrit.wikimedia.org/r/888052

I think the above change should work, but I’d like to test it a bit more tomorrow before moving the task into peer review.

In T327514#8598366, @ItamarWMDE wrote:

• In order to protect against malicious queries (these should never be in real sitelink URLs), don’t decode (or, re-encode after decoding) stuff like
  • whitespace characters
  • control characters

Some of those characters are required by other scripts and therefore do appear in real sitelink URLs. Zero-width joiner and zero-width non-joiner in particular can be relatively common in Arabic and Indic scripts, and select * { ?sitelink schema:isPartOf <https://fa.wikisource.org/> } limit 1000 includes quite a few zero-width non-joiners.

Change 888052 merged by jenkins-bot:

[wikidata/query/gui@master] Decode URIs for display

https://gerrit.wikimedia.org/r/888052

Change 890778 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: WDQSGuiBuilder):

[wikidata/query/gui-deploy@production] Merging from 1f13e68d3c8e39589eef930efdcecef4f68bbd7a

https://gerrit.wikimedia.org/r/890778

Change 890778 merged by Lucas Werkmeister (WMDE):

[wikidata/query/gui-deploy@production] Merging from 1f13e68d3c8e39589eef930efdcecef4f68bbd7a

https://gerrit.wikimedia.org/r/890778

Should be deployed now:

In T327514#8630437, @Nikki wrote:

In T327514#8598366, @ItamarWMDE wrote:

• In order to protect against malicious queries (these should never be in real sitelink URLs), don’t decode (or, re-encode after decoding) stuff like
  • whitespace characters
  • control characters

Some of those characters are required by other scripts and therefore do appear in real sitelink URLs. Zero-width joiner and zero-width non-joiner in particular can be relatively common in Arabic and Indic scripts, and select * { ?sitelink schema:isPartOf <https://fa.wikisource.org/> } limit 1000 includes quite a few zero-width non-joiners.

Hm, true, this doesn’t look nice :/

Let me see if there’s a more restrictive Unicode category we can use.

Not really – ZWJ/ZWNJ are in Other, format (Cf) together with the directional control characters (U+202E RIGHT-TO-LEFT OVERRIDE and friends), which I don’t think we want to allow in decoded form.

MediaWiki core’s MediaWikiTitleCodec::splitTitleString() hard-codes the bidi characters as forbidden: U+200E-F and U+202A-E. I guess we could do the same, and re-encode those seven while allowing the rest of the Cf category? (But still blocking the other “other” categories: Cc Other, control; Cs Other, surrogate; Co Other, private use; and Cn, Other, not assigned.)

(MediaWiki allows the bidi isolate characters in titles, and indeed U+2066 is a working redirect on enwiki. I’m not sure how I feel about that tbh.)

In T327514#8632160, @Lucas_Werkmeister_WMDE wrote:

[...] MediaWiki core’s MediaWikiTitleCodec::splitTitleString() hard-codes the bidi characters as forbidden: U+200E-F and U+202A-E. I guess we could do the same, and re-encode those seven while allowing the rest of the Cf category? [...]

Could we maybe go the opposite way? Having an allow-list of characters in Cf that we explicitly decode? Then we could maybe start with ZWJ/ZWNJ and add further chars as needed. I imagine that this would feel safer and more understandable to me when reading the code.

In T327514#8632710, @Michael wrote:

Could we maybe go the opposite way? Having an allow-list of characters in Cf that we explicitly decode? Then we could maybe start with ZWJ/ZWNJ and add further chars as needed. I imagine that this would feel safer and more understandable to me when reading the code.

Maybe, though I felt like some of the other characters in that Cf list also looked like they might theoretically be useful (some of them are even printable). But we could also start with ZW(N)J, sure.

I discovered the "all-titles" dumps a few days ago and realised I could use them to find page names containing \p{C} or \p{Z}. (I've put the commands I used in P44829)

There are about 1.1 million page names with those characters:

• Almost all are \p{Cf}
• ~21k have unassigned characters (\p{Cn}, list: P44820) (some of these were assigned in Unicode 15, I probably need to upgrade something)
• ~10k have private-use area characters (\p{Co}, list: P44816)
• ~4k have control characters (\p{Cc}, list: P44812)
• There are no \p{Z} or \p{Cs}

Of the \p{Cf} characters:

• ~1 million have zero-width non-joiners
• ~30k have zero-width joiners (list: P44807)
• ~30k have zero-width spaces (list: P44806)
• ~2k have soft hyphens (list: P44801)
• ~1k have byte-order marks/zero-width non-breaking spaces (list: P44803)
• ~1k have word joiners (list: P44802)
• ~500 have tags (list: P44804)
• ~500 have other \p{Cf} characters (list: P44822)

Looking at that, I would definitely include zero-width space too.

The tag characters are used for some emoji flags and we do have some sitelinks in Wikidata which use them (Q65300420, Q100587671) but I think individual tag characters (when not part of a recognised sequence) are not considered printable characters, so it's probably better to not decode those.

Private-use area characters do appear in some sitelinks (e.g. Q33061193), but whether they display properly or not depends whether a compatible font is used, so there's probably limited benefit to decoding those.

Word joiner characters are the proper way to encode a zero-width non-breaking space (to prevent breaking at that point). I think those would be fine to decode.

Soft hyphens are used to indicate where long words can break and are used in some particularly long page names (e.g. Q101), so not decoding it is counter-productive (e.g. this query's results wouldn't scroll horizontally if the soft hyphen were decoded).

Alright, then let’s allowlist ZWJ, ZWNJ and ZWSP.

Change 892511 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[wikidata/query/gui@master] Decode ZWJ, ZWNJ and ZWSP for display

https://gerrit.wikimedia.org/r/892511

Change 892511 merged by jenkins-bot:

[wikidata/query/gui@master] Decode ZWJ, ZWNJ and ZWSP for display

https://gerrit.wikimedia.org/r/892511

Moving this to verification, as I suppose this is all we're going to do here in terms of edge case handling.

Change 892900 had a related patch set uploaded (by WDQSGuiBuilder; author: WDQSGuiBuilder):

[wikidata/query/gui-deploy@production] Merging from b6a19cca3172f192f51d8d9552a7eca073d9b59f

https://gerrit.wikimedia.org/r/892900

Change 892900 merged by Lucas Werkmeister (WMDE):

[wikidata/query/gui-deploy@production] Merging from b6a19cca3172f192f51d8d9552a7eca073d9b59f

https://gerrit.wikimedia.org/r/892900