MediaWiki Developer stores different HTML output for different language variants in the parser cache
Closed, DuplicatePublic
Actions

Assigned To

None

Authored By

	• eprodromou
	Sep 10 2020, 7:51 PM

Description

"As a MediaWiki Developer, I want to store different HTML output for different language variants in the parser cache, so that I can return the right formatting for the user to read."

This is for handling things like traditional Chinese characters vs. simplified Chinese characters. I have to admit that I'm not 100% sure how this works in RESTBase, but it's something that's important there so we should consider it for this work.

I think it's worthwhile to have this discussion on one ticket, so let's say that for this user story we need to make sure we cover:

get
set
purge

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Support language variants in /page/html handler.	mediawiki/core	master	+540 -137
	Support language variants in /page/html handler.	mediawiki/core	master	+532 -146

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Stalled	None	T324931 Clean up open RESTBase related tickets
Open	None	T262315 <CORE TECHNOLOGY> API Migration & RESTBase Sunset
Resolved	MSantos	T264669 Move VE API from RESTBase into core.
Invalid	None	T262572 Use Parsoid-specific Parser Cache for Visual Editor
Resolved	Dbrant	T328943 Replace PCS lazy-loading logic with standard "loading=lazy" attribute
Resolved	MSantos	T314025 [EPIC] Migrate PCS service away from restbase
Open	None	T321207 Core REST endpoints should cover needs of node services
Resolved	abi_	T317019 REST page/html endpoints should support language variants
Duplicate	None	T262593 MediaWiki Developer stores different HTML output for different language variants in the parser cache

Event Timeline

• eprodromou created this task.Sep 10 2020, 7:51 PM

• eprodromou updated the task description. (Show Details)

ssastry moved this task from Needs Triage to Non-Parsing-Team Tasks on the Parsoid board.Sep 11 2020, 5:28 PM

• eprodromou removed a project: Platform Engineering.Sep 15 2020, 8:10 PM

Shizhao added a project: Chinese-Sites.Sep 16 2020, 3:06 AM

Ok, in Mediawiki ParserCache this theoretically should be coming out of the box, since userlang is a cache varying parser option and I believe it will vary on lang variants. But have to double-check.

• eprodromou triaged this task as Medium priority.Sep 30 2020, 1:35 PM

• eprodromou edited projects, added Platform Team Workboards (Green); removed Platform Team Workboards (User Stories).

• eprodromou moved this task from Backlog to Next Sprint on the Platform Team Workboards (Green) board.Sep 30 2020, 1:39 PM

• eprodromou edited parent tasks, added: T262572: Use Parsoid-specific Parser Cache for Visual Editor; removed: T262571: Parser Cache Support for Multiple Parsers.Oct 21 2020, 4:15 PM

• Pchelolo claimed this task.Oct 23 2020, 2:16 PM

• Pchelolo moved this task from Next Sprint to Doing on the Platform Team Workboards (Green) board.

Change 636070 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/core@master] WIP: Support language variants in /page/html handler.

https://gerrit.wikimedia.org/r/636070

gerritbot added a project: Patch-For-Review.Oct 23 2020, 5:04 PM

Change 636079 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/core@master] Support language variants in /page/html handler.

https://gerrit.wikimedia.org/r/636079

Change 636070 abandoned by Ppchelko:
[mediawiki/core@master] Support language variants in /page/html handler.

Reason:
git review operator error

https://gerrit.wikimedia.org/r/636070

In T262593#6485102, @Pchelolo wrote:

Ok, in Mediawiki ParserCache this theoretically should be coming out of the box, since userlang is a cache varying parser option and I believe it will vary on lang variants. But have to double-check.

No, the content variant is independent of the user interface language.

As far as I know, variant transformation is done on the fly, on top of cached content. So we wouldn't need or want ParserCache to know about it.

But let's talk to @Amire80 and @cscott to make sure.

No, the content variant is independent of the user interface language.

Then we can add one more parser option.

As far as I know, variant transformation is done on the fly, on top of cached content. So we wouldn't need or want ParserCache to know about it.

With the old parser, afaik - yes. Same is happening in RESTBase now. However, in RESTBase we are experiencing problems with on-the-fly conversion into lang variants - this coversion is quite resource-intensive, so from time to time we crash some app servers doing those conversions. The patch I made is caching the variants and all the intermediate transformations, which I think is the end goal we want to move towards.

In T262593#6579362, @Pchelolo wrote:

No, the content variant is independent of the user interface language.

Then we can add one more parser option.

Yes, we could do that.

As far as I know, variant transformation is done on the fly, on top of cached content. So we wouldn't need or want ParserCache to know about it.

With the old parser, afaik - yes. Same is happening in RESTBase now. However, in RESTBase we are experiencing problems with on-the-fly conversion into lang variants - this coversion is quite resource-intensive, so from time to time we crash some app servers doing those conversions. The patch I made is caching the variants and all the intermediate transformations, which I think is the end goal we want to move towards.

That sounds reasonable, but I'd want to first understand why we didn't go that route in the past. Also, this is not something I'd want to fold into the current initiative.

• WDoranWMF moved this task from Doing to Ready on the Platform Team Workboards (Green) board.Nov 18 2020, 5:20 PM

MBinder_WMF edited projects, added Parsoid (Tracking); removed Parsoid.Dec 10 2020, 8:12 PM

daniel moved this task from Ready to Backlog on the Platform Team Workboards (Green) board.Dec 16 2020, 5:20 PM

daniel removed • Pchelolo as the assignee of this task.Jan 4 2021, 3:33 PM

Winston_Sung subscribed.Nov 14 2021, 3:31 AM

Winston_Sung moved this task from Backlog to MediaWiki core on the Chinese-Sites board.Nov 14 2021, 3:49 AM

• Pchelolo mentioned this in T264669: Move VE API from RESTBase into core..Feb 9 2022, 3:53 PM

daniel mentioned this in T308743: PageHTMLHandler: add support for getting fully annotated parsoid HTML..Aug 4 2022, 10:15 AM

daniel added a parent task: T317019: REST page/html endpoints should support language variants.Sep 5 2022, 9:44 AM

Restricted Application added a subscriber: Ericliu1912. · View Herald TranscriptSep 5 2022, 9:44 AM

Pppery edited projects, added Patch-Needs-Improvement; removed Patch-For-Review.Oct 25 2023, 4:19 AM

We will be tracking it separately in T380517: Make Parsoid language conversion into an OutputTransform pass.