Maniphest T393391

Refactor PEG grammar for transclusions
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	cscott
	May 5 2025, 2:40 PM

Tags

Referenced Files

None

Subscribers

Jdforrester-WMF

Mr._Starfleet_Command

Description

We don't actually need to tokenize inside transclusions, just find the end. This should improve performance.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Bump wikimedia/parsoid to 0.22.0-a28	mediawiki/vendor	master	+12 K -2 K
	New PreprocTk tokens for "unexpanded preprocessed text"	mediawiki/services/parsoid	master	+11 K -2 K
	Curly-brace tokenizer: handle <noinclude>, <includeonly>, and <onlyinclude>	mediawiki/services/parsoid	master	+4 K -2 K

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open	Feature	None	T72077 Order of named parameters passed to parser functions is lost
		Open		None	T204307 Parser Functions should support named parameters
		Open		None	T388786 Follow up from Parsoid Fragment support
		Resolved		cscott	T390344 v3 parserfunction serialization doesn't properly support named arguments
		Open		cscott	T395946 PEG backtracking causes OOMs in some cases
		Resolved		cscott	T408631 Parsoid renders {{legislationuk}} incorrectly
		Open		None	T420060 Tokenizer: Bad markup in template arg prevents entire transclusion from being recognized
		Open		None	T394836 Use refactored grammar for `{{....}}` constructs
		Open		cscott	T395956 Use refactored grammar for `[..]` and `[[...]]` constructs
		Resolved		cscott	T394834 Use refactored PEG grammar for `{{#....}}` constructs
		Resolved		cscott	T393391 Refactor PEG grammar for transclusions

Event Timeline

cscott created this task.May 5 2025, 2:40 PM

cscott edited projects, added Parsoid-Read-Views (Performance), Content-Transform-Team (Work In Progress); removed Parsoid-Read-Views (Phase 3 - Main namespace of officewiki / mediawiki.org renders with Parsoid).

Change #1140966 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] WIP: new curly-brace tokenizer based on the preprocessor

https://gerrit.wikimedia.org/r/1140966

gerritbot added a project: Patch-For-Review.May 5 2025, 2:42 PM

cscott mentioned this in T393306: Chart output makes VE support difficult.May 5 2025, 3:08 PM

cscott mentioned this in T393726: Cache WikiLink processing in WikiLinkHandler.May 8 2025, 8:41 PM

cscott moved this task from Backlog to In Progress on the Content-Transform-Team (Work In Progress) board.May 12 2025, 3:21 PM

cscott mentioned this in T394834: Use refactored PEG grammar for `{{#....}}` constructs.May 20 2025, 7:09 PM

cscott renamed this task from Refactor PEG grammar for transclusion arguments to Refactor PEG grammar for transclusions.May 20 2025, 7:13 PM

MSantos moved this task from Needs Triage to Tech Debt / Big changes on the Parsoid board.May 22 2025, 2:31 PM

ssastry assigned this task to cscott.May 22 2025, 5:33 PM

ssastry triaged this task as Medium priority.

cscott mentioned this in T306679: (Performance) Excessive backtracking processing template markup.May 22 2025, 5:39 PM

cscott mentioned this in T392939: AttributeExpander::any could be calling PipelineUtils::expandAttrValueToDOM with repeating content.May 22 2025, 5:57 PM

cscott mentioned this in T273817: WMFTimeoutException for page views on dewiki (due to bracket mismatch on a wiki link).May 22 2025, 6:18 PM

cscott mentioned this in T394836: Use refactored grammar for `{{....}}` constructs.Jun 3 2025, 7:44 PM

cscott moved this task from In Progress to Blocked on the Content-Transform-Team (Work In Progress) board.Aug 18 2025, 3:32 PM

cscott moved this task from Blocked to In Progress on the Content-Transform-Team (Work In Progress) board.Sep 15 2025, 3:16 PM

Change #1189601 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] Curly-brace tokenizer: handle <noinclude>, <includeonly>, and <onlyinclude>

https://gerrit.wikimedia.org/r/1189601

Change #1189601 abandoned by C. Scott Ananian:

[mediawiki/services/parsoid@master] Curly-brace tokenizer: handle <noinclude>, <includeonly>, and <onlyinclude>

Reason:

Squashed into I8c7b535d8424fa70ec4d813118d18479cfd12203

https://gerrit.wikimedia.org/r/1189601

Change #1140966 merged by jenkins-bot:

[mediawiki/services/parsoid@master] New PreprocTk tokens for "unexpanded preprocessed text"

https://gerrit.wikimedia.org/r/1140966

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2025, 10:31 PM

MSantos added a project: OKR-Work.Oct 20 2025, 2:05 PM

MSantos moved this task from In Progress to Code Review on the Content-Transform-Team (Work In Progress) board.Oct 20 2025, 3:18 PM

Change #1197319 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.22.0-a28

https://gerrit.wikimedia.org/r/1197319

gerritbot added a project: Patch-For-Review.Oct 20 2025, 6:16 PM

Change #1197319 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.22.0-a28

https://gerrit.wikimedia.org/r/1197319

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2025, 8:32 PM

cscott moved this task from Code Review to To Verify on the Content-Transform-Team (Work In Progress) board.Oct 27 2025, 3:25 PM

cscott removed a subtask: T395956: Use refactored grammar for `[..]` and `[[...]]` constructs.Oct 27 2025, 4:54 PM

cscott added a parent task: T395956: Use refactored grammar for `[..]` and `[[...]]` constructs.

cscott removed a subtask: T394836: Use refactored grammar for `{{....}}` constructs.

cscott added a parent task: T394836: Use refactored grammar for `{{....}}` constructs.

cscott removed a subtask: T394834: Use refactored PEG grammar for `{{#....}}` constructs.Oct 27 2025, 4:56 PM

cscott added a parent task: T394834: Use refactored PEG grammar for `{{#....}}` constructs.

cscott closed this task as Resolved.Nov 3 2025, 4:14 PM

Mr._Starfleet_Command subscribed.Feb 9 2026, 10:10 PM

cscott mentioned this in T395946: PEG backtracking causes OOMs in some cases.Feb 16 2026, 2:29 AM