We have two options here:

1. Keep the ParserOutput unchanged, but ensure that skins property convert the data from ParserOutput::getSections() before using it. The proper conversion language is already present in the ParserOutput. This avoids changing the result returned by various APIs which already expose ParserOutput::getSections(), but perhaps makes it more likely that clients will implement this wrong (or forget to implement it).
2. Do language conversion on the getSections data *before* storing it in the ParserOutput. On the positive side, the skin doesn't need to explicitly handle conversion. On the negative side, this changes the results returned by various MediaWiki APIs which have returned getSections data for years now, potentially breaking downstream code.

Parsoid complicates the scenario somewhat because language conversion happens as a separate post processing pass in Parsoid. But there's no reason why parsoid's post processing can't also include the section metadata stored in ParserOutput. So probably either 1 or 2 would work fine with Parsoid as well.

I lean toward option 2, because it means that clients don't have to explicitly think about conversion, it 'just works'. But there is a slight risk that this will break clients, who if they'd been manually converting section data before might end up double-converting titles.

Change 787842 abandoned by Func:

[mediawiki/skins/Vector@master] SkinVector22: Apply language converter on section titles

Reason:

https://gerrit.wikimedia.org/r/787842

In T306862#8155609, @cscott wrote:

2. Do language conversion on the getSections data *before* storing it in the ParserOutput. On the positive side, the skin doesn't need to explicitly handle conversion. On the negative side, this changes the results returned by various MediaWiki APIs which have returned getSections data for years now, potentially breaking downstream code.

I want to make sure if this solution can solve T303855: Page-wide language conversion rules are ignored in the ToC which @Winston_Sung also mentioned before in T306862#8030445 with different examples?

I would like to fix the output of ParserOutput::getSections() to let the function itself return the expected converted result, but have no idea how to.

I agree with @Winston_Sung (that's "option 2" above), we should probably be a bit careful about it to make sure we don't break clients. This is somewhat related to T315222 which identifies a different bug in getSections() output that could/should be fixed.

And I agree with @Func that this is probably related to T303855, in that conversion of both the TOCHTML and the getSections() data should probably be applied before the contents are put into the ParserOutput.

Reaffirming above comment: the language conversion of the section data and the section HTML needs to be moved to the end of the parse, instead of being done in ParserOutput::getText().

@cscott is targetting the week of 19th to fix this one at which point web team may want to QA. C Scott will likely get someone from the language team to look at it in Subbu’s absence.

Change 836854 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] WIP fix lang conv of sections

https://gerrit.wikimedia.org/r/836854

Sorry, I haven't gotten this patch finished yet, but it's still on my radar.

In T306862#8454391, @cscott wrote:

Sorry, I haven't gotten this patch finished yet, but it's still on my radar.

Hey @cscott - do you have a rough estimate for when we can expect this to be wrapped up? It's currently a blocker for us, as we're trying to deploy Vector 2022 across all Wikipedias, and this is blocking the deployment to most Chinese sites.

Change 881937 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] Language-convert Table of Contents at parse time

https://gerrit.wikimedia.org/r/881937

Change 884084 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] Remove back-compatibility NO_TOC_CONVERSION code

https://gerrit.wikimedia.org/r/884084

Change 881937 merged by jenkins-bot:

[mediawiki/core@master] Language-convert Table of Contents at parse time

https://gerrit.wikimedia.org/r/881937

@ovasileva @Edtadros perhaps web team can QA this?

This should ride the train this week; added User-notice to get a mention on tech news. If someone could post in the zhwiki village pump that would be helpful!

Great, thanks!

I verified visually the characters

AC passed.

@Edtadros As a Chinese speaker I can confirm it is correctly converted.

@Jdlrobson The example provided does not contain in-article conversion rules, which should also be QA-ed due to wide usage.

@Diskdance would you be able to edit the description and the beta cluster page to add QA steps for those? I agree we should QA anything with wide usage and I am not confident in myself in covering them all.

Thanks @Winston_Sung for the confirmation on the screenshots! Much appreciated!

Would this be backported to 1.39 since it is arguably a regression with the ToC?

@Jdlrobson @Edtadros you don't need to have knowledge of the Chinese language to QA this. This variant converter bug only affects skins that generate ToC through Mustache template data (e.g. Vector 2022, Citizen, etc). You can compare the content of the ToC to skins that does not use Mustache for ToC (e.g. Timeless, Monobook, Modern, and most other skins).

I am not sure where can I find the QA template so hope the following format works:

QA Steps

1. Open the article in Vector 2022
2. Open the article in Timeless (by adding useskin=timeless to the end of the URL)
3. Switch to a language variant (at least compare simplified (简体) and traditional (繁體))
4. Compare the ToC titles (✅ if they are identical)

EDIT: Just realized that Legacy Vector also uses the old ToC. In that case using Legacy Vector for comparison also works.

In T306862#8649374, @cscott wrote:

This should ride the train this week; added User-notice to get a mention on tech news. [...]

Hi, for Tech News, what wording would you suggest as the content? (Drafts/summary-details always help!) - From reading all the comments above, I'm still unsure of details, and it might be good to include in the entry:

• what was the fundamental problem(s)?
• where did it affect? (E.g. is this problem related to only wikis using Chinese languages, or to all wikis that use multiple writing systems?)
• do any volunteers need to do anything? (or is this just informational?)
• is it already fixed (so belongs in "Recent changes" section) or on the deployment train ("Changes later this week" section)?

My current best-guess is along the lines of:

• Recent changes:
  • Chinese language wikis had problems with the language converter (used for multiple writing systems) when used in the site sidebar and table of contents. This has now been fixed.

Something more accurate, would be appreciated. Thanks.

在T306862#8655255中，@Winston_Sung写道：

I verified visually the characters

AC passed.

Me too

在T306862#8662356中，@Quiddity写道：

My current best-guess is along the lines of:

• Recent changes:
  • Chinese language wikis had problems with the language converter (used for multiple writing systems) when used in the site sidebar and table of contents. This has now been fixed.

Something more accurate, would be appreciated. Thanks.

thx, I update to https://meta.wikimedia.org/wiki/Tech/News/2023/10

Just curious, can this task be closed without a proper QA? We can confirm it works as expected on Chinese Wikipedia.

@Edtadros - could you take a look and add some screenshots as well?

I have created https://zh.wikipedia.org/wiki/User:Diskdance/public/T306862_test for testing, and found this inconsistency (same in Vector 2022):

The patch description says ToC use all page-wide rules so I believe this is expected, but I would like to confirm it with @cscott.

@Shizhao I believe this should also be mentioned in Tech News so users won't accidentally write things like this.

@Diskdance That does seem like expected behavior. We'd like to suggest that "best style" is to ensure all conversion rules are at the top of the document (ideally in glossaries, as is typical practice on zhwiki I believe), which avoids this particular discrepancy.

@Shizhao I don't think {{DISPLAYTITLE}} is related to this task (or this patch set) in one way or the other, but there was earlier work on displaytitle which might be involved: eed3121a8fca38bbd523c7068087e611f2606bb5 / e085e3f310b201c97c265cd160b321d146097a68 / bacd87e4942baa34808a1b77d3b29bfdb566cc17 / T67747: Sanitizer::removeHTMLtags doesn't close tags correctly when $wgUseTidy is enabled. Needs further investigation: I filed T331316: Vector (or Vector 2022?) displays {{DISPLAYTITLE}} only when no ToC is present for follow-up.

Change 894746 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] WIP: Don't clear LanguageConverter display title when converting ToC

https://gerrit.wikimedia.org/r/894746

[mediawiki/core@master] WIP: Don't clear LanguageConverter display title when converting ToC

@cscott perhaps we could capture the remaining work in a new ticket?

In T306862#8671076, @Jdlrobson wrote:

[mediawiki/core@master] WIP: Don't clear LanguageConverter display title when converting ToC

@cscott perhaps we could capture the remaining work in a new ticket?

Yes, already done: T331316. This ticket can be closed.

Change 894746 merged by jenkins-bot:

[mediawiki/core@master] Don't clear LanguageConverter display title when converting ToC

https://gerrit.wikimedia.org/r/894746

Change 884084 merged by jenkins-bot:

[mediawiki/core@master] Parser: Remove back-compatibility NO_TOC_CONVERSION code

https://gerrit.wikimedia.org/r/884084