Page MenuHomePhabricator
Create Task
Maniphest T373384

Parsoid doesn't properly handle double-underscore magic words
Open, MediumPublic
Actions

Description

For the double-underscore magic words __HIDDENCAT__, __NOINDEX__, and __INDEX__, Parsoid is missing the tracking category corresponding to this magic word. This seems to indicate that Parsoid is not actually invoking the legacy parser to handle these magic words, and so the resulting ParserOutput is likely to be omitting the proper index policy values as well.

The __NOGALLERY__ and __NOTOC__ magic words are handled in the same place of the legacy parser, and might also be missing some necessary behavior flags.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
ParsoidParser: set index/noindex in ParserOutputC. Scott Ananianmediawiki/coremaster+107 -6
Bump wikimedia/parsoid to 0.24.0-a2C. Scott Ananianmediawiki/vendorwmf/1.47.0-wmf.1+14 K -14 K
Bump wikimedia/parsoid to 0.24.0-a2C. Scott Ananianmediawiki/vendormaster+14 K -14 K
Set page properties corresponding to magic word behavior switchesC. Scott Ananianmediawiki/services/parsoidmaster+19 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

cscott created this task.Aug 26 2024, 6:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2024, 6:51 PM
MSantos assigned this task to cscott.Aug 29 2024, 2:28 PM
MSantos added a project: Content-Transform-Team-WIP.
MSantos moved this task from Uncategorized to DiscussionTools on the Parsoid-Read-Views board.
MSantos edited projects, added: Parsoid-Read-Views (DiscussionTools); removed: Parsoid-Read-Views.
MSantos moved this task from Needs Triage to Missing Functionality on the Parsoid board.Sep 19 2024, 2:25 PM
cscott added a comment.Sep 20 2024, 6:29 AM

We have Env::{get,set}BehaviorSwitch in Parsoid, and this potentially violates fragment independence as well when we interrogate the state of a number of these behavior switches in src/Wt2Html/DOM/Processors/WrapSectionsState.

We should probably be invoking some code in SiteConfig to handle the switches to ensure that we don't drift from the core implementation, although we can probably handle it in Parsoid in src/Wt2Html/TT/BehaviorSwitchHandler.

ihurbain moved this task from DiscussionTools to Small Size Wikipedias on the Parsoid-Read-Views board.Apr 10 2025, 12:53 PM
ihurbain edited projects, added: Parsoid-Read-Views (Small Size Wikipedias); removed: Parsoid-Read-Views (DiscussionTools).
ihurbain removed a project: Content-Transform-Team-WIP.
cscott added a project: Content-Platform-Team (Work In Progress).Jul 10 2025, 2:49 PM
cscott moved this task from Backlog to Q1 FY26-27 on the Content-Platform-Team (Work In Progress) board.
cscott added a subtask: T406186: Parsoid doesn't error when display title overrides earlier display title.Oct 2 2025, 2:27 PM
cscott mentioned this in T406186: Parsoid doesn't error when display title overrides earlier display title.Oct 2 2025, 2:29 PM
MSantos added a project: OKR-Work.Oct 20 2025, 2:08 PM
cscott closed subtask T407290: Parsoid doesn't use the list of double-underscore magic words from MagicWordFactory::getDoubleUnderscoreArray() as Resolved.Dec 2 2025, 3:50 AM
cscott closed subtask T407289: Parsoid doesn't handle Japanese behavior switches with U+FF3F (full width underscore) as Resolved.
MSantos triaged this task as Medium priority.Jan 8 2026, 3:14 PM
MSantos moved this task from Q1 FY26-27 to Backlog on the Content-Platform-Team (Work In Progress) board.
cscott mentioned this in T424773: __NOCONTENTCONVERT__ is not honored in Parsoid.Apr 28 2026, 10:49 PM
cscott added a parent task: T424773: __NOCONTENTCONVERT__ is not honored in Parsoid.
gerritbot added a comment.Apr 29 2026, 7:44 PM

Change #1279469 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] Set page properties corresponding to magic word behavior switches

https://gerrit.wikimedia.org/r/1279469

gerritbot added a project: Patch-For-Review.Apr 29 2026, 7:44 PM
gerritbot added a comment.Apr 29 2026, 9:27 PM

Change #1279495 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] ParsoidParser: set index/noindex in ParserOutput

https://gerrit.wikimedia.org/r/1279495

gerritbot added a comment.May 1 2026, 4:43 AM

Change #1279469 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Set page properties corresponding to magic word behavior switches

https://gerrit.wikimedia.org/r/1279469

gerritbot added a comment.May 7 2026, 7:44 PM

Change #1284813 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.24.0-a2

https://gerrit.wikimedia.org/r/1284813

gerritbot added a comment.May 7 2026, 8:02 PM

Change #1284813 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.24.0-a2

https://gerrit.wikimedia.org/r/1284813

gerritbot added a comment.May 7 2026, 8:16 PM

Change #1284832 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/vendor@wmf/1.47.0-wmf.1] Bump wikimedia/parsoid to 0.24.0-a2

https://gerrit.wikimedia.org/r/1284832

gerritbot added a comment.May 7 2026, 9:07 PM

Change #1284832 merged by jenkins-bot:

[mediawiki/vendor@wmf/1.47.0-wmf.1] Bump wikimedia/parsoid to 0.24.0-a2

https://gerrit.wikimedia.org/r/1284832

Stashbot mentioned this in T368724: Rendering diff on broken link with template (visual diff testing).May 7 2026, 9:15 PM
Stashbot mentioned this in T319058: ParserTests defaults to wgAllowExternalImages=true.
Stashbot mentioned this in T424446: Space added for no reason.
Stashbot mentioned this in T420336: mw-parsoid improvements.
Stashbot mentioned this in T425008: WrapSectionState crash: Wikimedia\Assert\InvariantException: Invariant failed: Expected only language variants to be missing about ids..
Stashbot mentioned this in T425107: __TOC__ not showing up in Parsoid on Norwegian Bokmål village pump pages.
Stashbot mentioned this in T425056: 1.46.0-wmf.26 broke itwiki's Template:Divisa calcio.
Stashbot mentioned this in T425731: CTT tasks week of 2026-05-01.

Mentioned in SAL (#wikimedia-operations) [2026-05-07T21:15:34Z] <cscott@deploy1003> Started scap sync-world: Backport for [[gerrit:1284828|Upgrading webonyx/graphql-php (v15.31.5 => v15.32.3)]], [[gerrit:1284834|composer.json: Update webonyx/graphql-php to ^15.32.3]], [[gerrit:1284832|Bump wikimedia/parsoid to 0.24.0-a2 (T319058 T368724 T373384 T420336 T423241 T423701 T424446 T424773 T425008 T425056 T425107 T425731)]], [[gerrit:1284837|Bump wikimedia/parsoid to 0.24.0-a2 (T425731)]

Stashbot mentioned this in T423701: Serialize ContentHolder (or at least its fragments) in ParserOutput.May 7 2026, 9:16 PM
Stashbot mentioned this in T423241: Thumbtime for TimedMedia no longer working.

Mentioned in SAL (#wikimedia-operations) [2026-05-07T21:17:28Z] <cscott@deploy1003> cscott: Backport for [[gerrit:1284828|Upgrading webonyx/graphql-php (v15.31.5 => v15.32.3)]], [[gerrit:1284834|composer.json: Update webonyx/graphql-php to ^15.32.3]], [[gerrit:1284832|Bump wikimedia/parsoid to 0.24.0-a2 (T319058 T368724 T373384 T420336 T423241 T423701 T424446 T424773 T425008 T425056 T425107 T425731)]], [[gerrit:1284837|Bump wikimedia/parsoid to 0.24.0-a2 (T425731)]] synced to the t

Stashbot added a comment.May 7 2026, 9:27 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-07T21:27:35Z] <cscott@deploy1003> Finished scap sync-world: Backport for [[gerrit:1284828|Upgrading webonyx/graphql-php (v15.31.5 => v15.32.3)]], [[gerrit:1284834|composer.json: Update webonyx/graphql-php to ^15.32.3]], [[gerrit:1284832|Bump wikimedia/parsoid to 0.24.0-a2 (T319058 T368724 T373384 T420336 T423241 T423701 T424446 T424773 T425008 T425056 T425107 T425731)]], [[gerrit:1284837|Bump wikimedia/parsoid to 0.24.0-a2 (T425731)

gerritbot added a comment.May 22 2026, 9:51 PM

Change #1279495 merged by jenkins-bot:

[mediawiki/core@master] ParsoidParser: set index/noindex in ParserOutput

https://gerrit.wikimedia.org/r/1279495

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.4; 2026-05-26).May 22 2026, 10:00 PM
Maintenance_bot removed a project: Patch-For-Review.May 22 2026, 10:31 PM
cscott mentioned this in T424370: [SR] Language Converter should run only on pages in main namespace.Jun 21 2026, 4:43 PM
cscott mentioned this in T349137: Handle __NOCONTENTCONVERT__ in core, not Parsoid.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits