Page MenuHomePhabricator
Create Task
Maniphest T407289

Parsoid doesn't handle Japanese behavior switches with U+FF3F (full width underscore)
Closed, ResolvedPublic
Actions

Description

Parsoid assumes all behavior switches start with __, which isn't the case in practice. We should deprecate behavior switches which don't start with __ or U+FF3F U+FF3F (the full width equivalent) and add support to Parsoid for handling behavior switches which start with U+FF3F

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Bump wikimedia/parsoid to 0.23.0-a6mediawiki/vendormaster+27 K -20 K
Get the list of behavior switch magic words from coremediawiki/services/parsoidmaster+25 K -18 K
i18n: Remove deprecated aliases for double-underscore magic wordsmediawiki/coremaster+43 -19
i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vepmediawiki/coreREL1_39+4 -4
i18n: deprecate double-underscore magic words which don't start/end with __mediawiki/coreREL1_39+323 -84
i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vepmediawiki/coreREL1_43+4 -4
i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vepmediawiki/coreREL1_44+4 -4
i18n: deprecate double-underscore magic words which don't start/end with __mediawiki/coreREL1_44+56 -37
i18n: deprecate double-underscore magic words which don't start/end with __mediawiki/coreREL1_43+56 -37
parser: Ensure that double-underscore magic words actually begin with `__`mediawiki/coremaster+36 -0
i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vepmediawiki/coreREL1_45+4 -4
i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vepmediawiki/corewmf/1.45.0-wmf.25+4 -4
i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vepmediawiki/coremaster+4 -4
Bump wikimedia/parsoid to 0.22.0mediawiki/vendorREL1_45+11 K -572
Support U+FF3F as behavior switch prefix, as used on jawikimediawiki/services/parsoidmaster+11 K -545
Add "double wide underscore" to wikitext escape listmediawiki/extensions/Scribuntomaster+13 -2
i18n: deprecate double-underscore magic words which don't start/end with __mediawiki/coremaster+60 -37
Add "double wide underscore" to wikitext escape listmediawiki/coremaster+18 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

cscott created this task.Oct 14 2025, 8:48 PM
gerritbot added a comment.Oct 14 2025, 8:56 PM

Change #1196178 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] Support U+FF3F as behavior switch prefix, as used on jawiki

https://gerrit.wikimedia.org/r/1196178

gerritbot added a project: Patch-For-Review.Oct 14 2025, 8:56 PM
gerritbot added a comment.Oct 14 2025, 9:42 PM

Change #1196126 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] parser: Ensure that double-underscore magic words actually begin with `__`

https://gerrit.wikimedia.org/r/1196126

gerritbot added a comment.Oct 14 2025, 9:42 PM

Change #1196182 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] i18n: all double-underscore magic words must start with double-underscore

https://gerrit.wikimedia.org/r/1196182

A_smart_kitten added projects: MediaWiki-Parser, MediaWiki-Internationalization.Oct 15 2025, 7:54 AM
Bugreporter2 renamed this task from Parsoid doesn't handle japanese behavior switches with U+FF3F (full width underscore) to Parsoid doesn't handle Japanese behavior switches with U+FF3F (full width underscore).Oct 15 2025, 12:37 PM
Bugreporter2 updated the task description. (Show Details)
gerritbot added a comment.Oct 15 2025, 9:26 PM

Change #1196527 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] Add "double wide underscore" to wikitext escape list

https://gerrit.wikimedia.org/r/1196527

MSantos moved this task from Backlog to In Progress on the Content-Transform-Team (Work In Progress) board.Oct 16 2025, 2:13 PM
MSantos added a project: OKR-Work.Oct 20 2025, 2:05 PM
gerritbot added a comment.Oct 24 2025, 2:32 AM

Change #1198419 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] WIP: Get the list of behavior switch magic words from core

https://gerrit.wikimedia.org/r/1198419

gerritbot added a comment.Oct 24 2025, 8:14 PM

Change #1198597 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/extensions/Scribunto@master] Add "double wide underscore" to wikitext escape list

https://gerrit.wikimedia.org/r/1198597

gerritbot added a comment.Oct 24 2025, 9:12 PM

Change #1198600 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] i18n: Remove deprecated aliases for double-underscore magic words

https://gerrit.wikimedia.org/r/1198600

gerritbot added a comment.Oct 25 2025, 10:43 PM

Change #1196527 merged by jenkins-bot:

[mediawiki/core@master] Add "double wide underscore" to wikitext escape list

https://gerrit.wikimedia.org/r/1196527

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.25; 2025-10-28).Oct 25 2025, 11:00 PM
gerritbot added a comment.Oct 25 2025, 11:41 PM

Change #1196182 merged by jenkins-bot:

[mediawiki/core@master] i18n: deprecate double-underscore magic words which don't start/end with __

https://gerrit.wikimedia.org/r/1196182

gerritbot added a comment.Oct 26 2025, 4:58 AM

Change #1198597 merged by jenkins-bot:

[mediawiki/extensions/Scribunto@master] Add "double wide underscore" to wikitext escape list

https://gerrit.wikimedia.org/r/1198597

cscott moved this task from In Progress to Code Review on the Content-Transform-Team (Work In Progress) board.Oct 27 2025, 3:26 PM
gerritbot added a comment.Oct 28 2025, 4:33 AM

Change #1196178 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Support U+FF3F as behavior switch prefix, as used on jawiki

https://gerrit.wikimedia.org/r/1196178

gerritbot added a comment.Oct 29 2025, 2:31 PM

Change #1199793 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/vendor@REL1_45] Bump wikimedia/parsoid to 0.22.0

https://gerrit.wikimedia.org/r/1199793

gerritbot added a comment.Oct 29 2025, 2:41 PM

Change #1199793 merged by C. Scott Ananian:

[mediawiki/vendor@REL1_45] Bump wikimedia/parsoid to 0.22.0

https://gerrit.wikimedia.org/r/1199793

gerritbot added a comment.Oct 30 2025, 3:19 PM

Change #1200091 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1200091

gerritbot added a comment.Oct 31 2025, 3:34 PM

Change #1200091 merged by jenkins-bot:

[mediawiki/core@master] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1200091

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.1; 2025-11-05).Oct 31 2025, 4:00 PM
gerritbot added a comment.Nov 3 2025, 2:09 PM

Change #1201070 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@wmf/1.45.0-wmf.25] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201070

gerritbot added a comment.Nov 3 2025, 2:53 PM

Change #1201070 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.25] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201070

Stashbot added a comment.Nov 3 2025, 2:53 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-03T14:53:50Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1201069|i18n: all behavior switches should start/end with __ (part 2)]], [[gerrit:1201070|i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep (T407289)]]

Stashbot added a comment.Nov 3 2025, 2:56 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-03T14:55:58Z] <lucaswerkmeister-wmde@deploy2002> cscott, lucaswerkmeister-wmde: Backport for [[gerrit:1201069|i18n: all behavior switches should start/end with __ (part 2)]], [[gerrit:1201070|i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep (T407289)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Nov 3 2025, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-03T15:03:36Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1201069|i18n: all behavior switches should start/end with __ (part 2)]], [[gerrit:1201070|i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep (T407289)]] (duration: 09m 45s)

gerritbot added a comment.Nov 3 2025, 3:48 PM

Change #1201091 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@REL1_45] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201091

gerritbot added a comment.Nov 3 2025, 4:07 PM

Change #1201091 merged by jenkins-bot:

[mediawiki/core@REL1_45] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201091

ReleaseTaggerBot edited projects, added MW-1.45-notes; removed MW-1.45-notes (1.45.0-wmf.25; 2025-10-28).Nov 3 2025, 5:00 PM
gerritbot added a comment.Nov 3 2025, 6:02 PM

Change #1196126 abandoned by C. Scott Ananian:

[mediawiki/core@master] parser: Ensure that double-underscore magic words actually begin with `__`

Reason:

Abandoned in favor of I8dd522d605c2b9e5310d169d7c51bcf424089497

https://gerrit.wikimedia.org/r/1196126

gerritbot added a comment.Nov 3 2025, 6:03 PM

Change #1201166 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@REL1_44] i18n: deprecate double-underscore magic words which don't start/end with __

https://gerrit.wikimedia.org/r/1201166

gerritbot added a comment.Nov 3 2025, 6:08 PM

Change #1201169 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@REL1_44] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201169

gerritbot added a comment.Nov 3 2025, 6:10 PM

Change #1201170 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@REL1_43] i18n: deprecate double-underscore magic words which don't start/end with __

https://gerrit.wikimedia.org/r/1201170

gerritbot added a comment.Nov 3 2025, 6:12 PM

Change #1201172 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@REL1_43] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201172

gerritbot added a comment.Nov 3 2025, 6:15 PM

Change #1201174 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@REL1_39] i18n: deprecate double-underscore magic words which don't start/end with __

https://gerrit.wikimedia.org/r/1201174

gerritbot added a comment.Nov 3 2025, 6:26 PM

Change #1201170 merged by jenkins-bot:

[mediawiki/core@REL1_43] i18n: deprecate double-underscore magic words which don't start/end with __

https://gerrit.wikimedia.org/r/1201170

gerritbot added a comment.Nov 3 2025, 6:28 PM

Change #1201166 merged by jenkins-bot:

[mediawiki/core@REL1_44] i18n: deprecate double-underscore magic words which don't start/end with __

https://gerrit.wikimedia.org/r/1201166

gerritbot added a comment.Nov 3 2025, 6:29 PM

Change #1201169 merged by jenkins-bot:

[mediawiki/core@REL1_44] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201169

gerritbot added a comment.Nov 3 2025, 6:36 PM

Change #1201180 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@REL1_39] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201180

gerritbot added a comment.Nov 3 2025, 6:40 PM

Change #1201172 merged by jenkins-bot:

[mediawiki/core@REL1_43] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201172

ReleaseTaggerBot added projects: MW-1.44-notes, MW-1.43-notes.Nov 3 2025, 7:00 PM
gerritbot added a comment.Nov 3 2025, 7:28 PM

Change #1201174 merged by jenkins-bot:

[mediawiki/core@REL1_39] i18n: deprecate double-underscore magic words which don't start/end with __

https://gerrit.wikimedia.org/r/1201174

gerritbot added a comment.Nov 3 2025, 7:29 PM

Change #1201180 merged by jenkins-bot:

[mediawiki/core@REL1_39] i18n: Remove deprecated behavior switches without underscores in et/sh-latn/vep

https://gerrit.wikimedia.org/r/1201180

ReleaseTaggerBot added a project: MW-1.39-notes.Nov 3 2025, 8:00 PM
cscott moved this task from Needs Triage to Bugs & Crashers on the Parsoid board.Nov 6 2025, 3:39 PM
gerritbot added a comment.Nov 11 2025, 1:52 AM

Change #1198600 merged by jenkins-bot:

[mediawiki/core@master] i18n: Remove deprecated aliases for double-underscore magic words

https://gerrit.wikimedia.org/r/1198600

ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.3; 2025-11-19); removed MW-1.46-notes (1.46.0-wmf.1; 2025-11-05).Nov 11 2025, 2:00 AM
gerritbot added a comment.Nov 14 2025, 9:31 PM

Change #1198419 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Get the list of behavior switch magic words from core

https://gerrit.wikimedia.org/r/1198419

Maintenance_bot removed a project: Patch-For-Review.Nov 14 2025, 10:30 PM
cscott moved this task from Code Review to To Deploy on the Content-Transform-Team (Work In Progress) board.Nov 17 2025, 4:18 PM
MSantos triaged this task as Medium priority.Nov 21 2025, 10:26 AM
gerritbot added a comment.Nov 24 2025, 10:15 PM

Change #1210725 had a related patch set uploaded (by Arlolra; author: Arlolra):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.23.0-a6

https://gerrit.wikimedia.org/r/1210725

gerritbot added a project: Patch-For-Review.Nov 24 2025, 10:15 PM
gerritbot added a comment.Nov 24 2025, 11:09 PM

Change #1210725 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.23.0-a6

https://gerrit.wikimedia.org/r/1210725

Maintenance_bot removed a project: Patch-For-Review.Nov 24 2025, 11:31 PM
ABreault-WMF moved this task from To Deploy to To Verify on the Content-Transform-Team (Work In Progress) board.Nov 28 2025, 1:28 AM
cscott closed this task as Resolved.Dec 2 2025, 3:50 AM
cscott mentioned this in T415405: Parsoid doesn't handle Japanese parser functions with double-wide hash and underscore.Fri, Jan 23, 8:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits