Page MenuHomePhabricator
Create Task
Maniphest T422966

Parsoid tokenizer doesn't allow table or list markup inside language conversion brackets
Closed, ResolvedPublic
Actions

Description

https://zh.wikipedia.org/wiki/%E6%B1%89%E8%AF%AD%E5%9C%B0%E5%8C%BA%E7%94%A8%E8%AF%8D%E5%B7%AE%E5%BC%82%E5%88%97%E8%A1%A8?useparsoid=0&variant=zh-hk

image.png (345×477 px, 71 KB)

vs
https://zh.wikipedia.org/wiki/%E6%B1%89%E8%AF%AD%E5%9C%B0%E5%8C%BA%E7%94%A8%E8%AF%8D%E5%B7%AE%E5%BC%82%E5%88%97%E8%A1%A8?useparsoid=1&variant=zh-hk&parsoidnewlc=1
image.png (467×477 px, 79 KB)

It appears that Parsoid's tokenizer doesn't allow table markup inside -{ }- brackets.

Another example, with list markup:
https://iu.wikipedia.org/w/index.php?title=%E1%90%8A%E1%95%8B%E1%95%95&useparsoid=0&variant=ike-cans
vs
https://iu.wikipedia.org/w/index.php?title=%E1%90%8A%E1%95%8B%E1%95%95&useparsoid=1&variant=ike-cans

image.png (306×509 px, 35 KB)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Bump wikimedia/parsoid to 0.23.0-a28mediawiki/vendorwmf/1.46.0-wmf.24+1 K -1 K
Bump wikimedia/parsoid to 0.23.0-a28mediawiki/vendormaster+1 K -1 K
Add heuristic about when to serialize lc content to newlinemediawiki/services/parsoidmaster+10 -12
Add test demonstrating wt2wt bug in language converter with listsmediawiki/services/parsoidmaster+37 -4
Allow language converter markup to contain tablesmediawiki/services/parsoidmaster+492 -287
Customize query in gerrit

Related Objects

Event Timeline

cscott created this task.Apr 10 2026, 6:10 PM
Restricted Application added subscribers: Stang, Aklapper. · View Herald TranscriptApr 10 2026, 6:10 PM
gerritbot added a comment.Wed, Apr 15, 12:33 PM

Change #1270114 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Allow language converter markup to contain tables

https://gerrit.wikimedia.org/r/1270114

cscott moved this task from Backlog to Code Review on the Content-Transform-Team (Work In Progress) board.Thu, Apr 16, 4:57 PM
gerritbot added a comment.Thu, Apr 16, 5:28 PM

Change #1271986 had a related patch set uploaded (by Arlolra; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] Add test demonstrating wt2wt bug in language converter with lists

https://gerrit.wikimedia.org/r/1271986

gerritbot added a project: Patch-For-Review.Thu, Apr 16, 5:28 PM

Change #1272804 had a related patch set uploaded (by Arlolra; author: Arlolra):

[mediawiki/services/parsoid@master] Add heuristic about when to serialize lc content to newline

https://gerrit.wikimedia.org/r/1272804

gerritbot added a comment.Thu, Apr 16, 7:13 PM

Change #1271986 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Add test demonstrating wt2wt bug in language converter with lists

https://gerrit.wikimedia.org/r/1271986

gerritbot added a comment.Thu, Apr 16, 7:15 PM

Change #1272804 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Add heuristic about when to serialize lc content to newline

https://gerrit.wikimedia.org/r/1272804

cscott renamed this task from Parsoid tokenizer doesn't allow table markup inside language conversion brackets to Parsoid tokenizer doesn't allow table or list markup inside language conversion brackets.Fri, Apr 17, 10:14 PM
cscott updated the task description. (Show Details)
cscott moved this task from Code Review to To Deploy on the Content-Transform-Team (Work In Progress) board.Fri, Apr 17, 11:33 PM
cscott mentioned this in T423785: Parsoid Read Views to deploy ~2026-04-20 (Language Converter wikis).Sat, Apr 18, 4:29 PM
gerritbot added a comment.Mon, Apr 20, 4:43 PM

Change #1275470 had a related patch set uploaded (by OSleger; author: OSleger):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.23.0-a28

https://gerrit.wikimedia.org/r/1275470

gerritbot added a comment.Mon, Apr 20, 10:09 PM

Change #1275470 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.23.0-a28

https://gerrit.wikimedia.org/r/1275470

gerritbot added a comment.Mon, Apr 20, 10:24 PM

Change #1275541 had a related patch set uploaded (by C. Scott Ananian; author: OSleger):

[mediawiki/vendor@wmf/1.46.0-wmf.24] Bump wikimedia/parsoid to 0.23.0-a28

https://gerrit.wikimedia.org/r/1275541

gerritbot added a comment.Tue, Apr 21, 2:05 PM

Change #1275541 merged by jenkins-bot:

[mediawiki/vendor@wmf/1.46.0-wmf.24] Bump wikimedia/parsoid to 0.23.0-a28

https://gerrit.wikimedia.org/r/1275541

Stashbot mentioned this in T420102: Special:LintTemplateErrors shows parser functions without context.Tue, Apr 21, 2:09 PM
Stashbot mentioned this in T423192: Galleries on Parsoid don't support data attributes.
Stashbot mentioned this in T423763: phpunit PKSA-5jz8-6tcw-pbk4 breaks CI.
Stashbot mentioned this in T423747: Parsoid LanguageConverter sets top level `lang` and `dir` attributes to the base language not the variant.
Stashbot mentioned this in T423662: CTT tasks week of 2026-04-17.

Mentioned in SAL (#wikimedia-operations) [2026-04-21T14:09:44Z] <cscott@deploy1003> Started scap sync-world: Backport for [[gerrit:1275541|Bump wikimedia/parsoid to 0.23.0-a28 (T420102 T421680 T422879 T422966 T423192 T423763 T423662)]], [[gerrit:1275542|Bump wikimedia/parsoid to 0.23.0-a28 (T423662)]], [[gerrit:1275560|[tests] add ParsoidLanguageConverterTest]], [[gerrit:1275561|ParsoidLanguageConverter: update lang/dir on content wrapper div (T423747)]]

Stashbot added a comment.Tue, Apr 21, 2:11 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-21T14:11:25Z] <cscott@deploy1003> cscott: Backport for [[gerrit:1275541|Bump wikimedia/parsoid to 0.23.0-a28 (T420102 T421680 T422879 T422966 T423192 T423763 T423662)]], [[gerrit:1275542|Bump wikimedia/parsoid to 0.23.0-a28 (T423662)]], [[gerrit:1275560|[tests] add ParsoidLanguageConverterTest]], [[gerrit:1275561|ParsoidLanguageConverter: update lang/dir on content wrapper div (T423747)]] synced to the testservers (see https://wikit

Stashbot added a comment.Tue, Apr 21, 2:22 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-21T14:22:46Z] <cscott@deploy1003> Finished scap sync-world: Backport for [[gerrit:1275541|Bump wikimedia/parsoid to 0.23.0-a28 (T420102 T421680 T422879 T422966 T423192 T423763 T423662)]], [[gerrit:1275542|Bump wikimedia/parsoid to 0.23.0-a28 (T423662)]], [[gerrit:1275560|[tests] add ParsoidLanguageConverterTest]], [[gerrit:1275561|ParsoidLanguageConverter: update lang/dir on content wrapper div (T423747)]] (duration: 13m 02s)

ABreault-WMF closed this task as Resolved.Fri, Apr 24, 12:56 AM
ABreault-WMF assigned this task to cscott.
ABreault-WMF moved this task from To Deploy to To Verify on the Content-Transform-Team (Work In Progress) board.
ABreault-WMF subscribed.

Post-deploy, these look the same now

https://zh.wikipedia.org/wiki/%E6%B1%89%E8%AF%AD%E5%9C%B0%E5%8C%BA%E7%94%A8%E8%AF%8D%E5%B7%AE%E5%BC%82%E5%88%97%E8%A1%A8?useparsoid=0&variant=zh-hk
https://zh.wikipedia.org/wiki/%E6%B1%89%E8%AF%AD%E5%9C%B0%E5%8C%BA%E7%94%A8%E8%AF%8D%E5%B7%AE%E5%BC%82%E5%88%97%E8%A1%A8?useparsoid=1&variant=zh-hk&parsoidnewlc=1

https://iu.wikipedia.org/w/index.php?title=%E1%90%8A%E1%95%8B%E1%95%95&useparsoid=0&variant=ike-cans
https://iu.wikipedia.org/w/index.php?title=%E1%90%8A%E1%95%8B%E1%95%95&useparsoid=1&variant=ike-cans

Stang unsubscribed.Mon, Apr 27, 8:33 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits