Page MenuHomePhabricator

Popups don‘t support language variant conversion and {{lang}} template
Closed, ResolvedPublic

Related Objects

StatusAssignedTask
ResolvedPchelolo
Resolvedmobrovac
ResolvedEevans
ResolvedEevans
ResolvedDzahn
ResolvedEevans
ResolvedEevans
OpenNone
DeclinedNone
ResolvedEevans
Resolvedfgiunchedi
ResolvedEevans
ResolvedPchelolo
Opencscott
OpenNone
Opencscott
Invalid GWicke
Resolvedliangent
Resolvedthiemowmde
OpenNone
Resolvedcscott
Resolvedcscott
ResolvedElitre
Resolvedcscott
Resolvedcscott
Resolvedcscott
Resolvedcscott
Resolvedcscott
Opencscott
Resolvedcscott
Opencscott
Opencscott
Opencscott
Resolvedmobrovac
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedEevans
ResolvedEevans
ResolvedEevans
Resolvedmobrovac
Resolvedcscott
ResolvedPchelolo
ResolvedPchelolo
ResolvedABorbaWMF

Event Timeline

Shizhao created this task.Feb 24 2018, 3:14 AM
Restricted Application added subscribers: Cosine02, Aklapper. · View Herald TranscriptFeb 24 2018, 3:14 AM
ovasileva triaged this task as High priority.Feb 26 2018, 12:03 PM
bearND added a subscriber: bearND.Feb 26 2018, 4:46 PM

This is a general RESTBase limitation T159985, which is blocked on Parsoid at the moment. On the Parsoid side there has been some progress lately.

@Shizhao can you provide some replication instructions? Although I'm starting to learn Chinese, my Chinese is not good enough to understand what the problem is here :).

From what I understand the title "《POP TEAM EPIC》" was not showing in the preview (and 《》 instead, but I see the following when testing:

. Did a local edit happen to fix this?

Could you provide a test case maybe at https://en.wikipedia.beta.wmflabs.org/wiki/T188164 and https://en.wikipedia.beta.wmflabs.org/wiki/Link_to_T188164 ?

@bearND any theories?

Shizhao added a comment.EditedFeb 27 2018, 3:08 AM

@Shizhao can you provide some replication instructions? Although I'm starting to learn Chinese, my Chinese is not good enough to understand what the problem is here :).
From what I understand the title "《POP TEAM EPIC》" was not showing in the preview (and 《》 instead, but I see the following when testing:

. Did a local edit happen to fix this?

Yes, local edit via other template fixed it https://zh.wikipedia.org/wiki/Template:Nihongo_title

test case in zhwiki: https://zh.wikipedia.org/wiki/User:Shizhao/T188164

and see https://zh.wikipedia.beta.wmflabs.org/wiki/T188164 https://zh.wikipedia.beta.wmflabs.org/wiki/Link_to_T188164

Liuxinyu970226 added a comment.EditedFeb 27 2018, 10:05 AM

@Shizhao and @Jdlrobson Unless if there's different configurations which I've not heard, contents in the first brackets are generally not shown (brackets are not shown too), e.g. by visiting Balloon loop and trying on tram or streetcar: "A tram is a rail vehicle which runs", where "(also tramcar; and in North America streetcar, trolley, trolley car, or cable car)" are ignored.

The problem regarding second brackets, where contents are also hidden but brackets shown, is interesting.

bearND updated the task description. (Show Details)Feb 27 2018, 10:42 PM
bearND added a comment.EditedFeb 27 2018, 10:53 PM

I've added the URL to the example page since it was not obvious to me from just looking at this task.

About the empty () in 《星色Girl Drop》():
The summary is derived from the Parsoid HTML. For this page this is https://zh.wikipedia.org/api/rest_v1/page/html/POP_TEAM_EPIC.
I was first thinking that the () was a remainder of removing some pronunciation but I now see that the Parsoid page has the same () in there.
In the wikitext I see a template for displaying Japanese words being used (Template:Nihongo: {{nihongo|《星色Girl Drop》|☆色ガールドロップ}}) . My current theory why the content between () is missing in the output is because Parsoid and RESTBase don't fully support language variants yet.

Mholloway added a project: Parsoid.EditedFeb 28 2018, 11:33 PM
Mholloway added a subscriber: Mholloway.

I'm not entirely convinced the empty parens issue is a language variant issue. It might just be a Parsoid bug. From what I understand Parsoid leverages action=expandtemplates to get template-generated content, and it expands this wikitext just fine:

https://en.wikipedia.org/w/api.php?action=expandtemplates&text={{nihongo|%E3%80%8A%E6%98%9F%E8%89%B2Girl%20Drop%E3%80%8B%7C%E2%98%86%E8%89%B2%E3%82%AC%E3%83%BC%E3%83%AB%E3%83%89%E3%83%AD%E3%83%83%E3%83%97}}

Tagging Parsoid for investigation. If it's a separate issue as I suspect then it should probably get its own task.

cscott added a subscriber: cscott.Mar 1 2018, 2:18 AM

The outer HTML for the "Girl Drop" section in https://zh.wikipedia.org/api/rest_v1/page/html/POP_TEAM_EPIC is

<span style="font-weight: normal;" about="#mwt31">(<span lang="ja"><span typeof="mw:LanguageVariant" data-mw-variant="{&quot;disabled&quot;:{&quot;t&quot;:&quot;☆色ガールドロップ&quot;}}"></span></span>)</span>

The blank section in the parens is the language variant markup. Your expandtemplates request is executing on en.wikipedia.org, instead of zh.wikipedia.org, so you're expanding the wrong template. This one contains the language converter markup:
https://zh.wikipedia.org/w/api.php?action=expandtemplates&text={{nihongo|%E3%80%8A%E6%98%9F%E8%89%B2Girl%20Drop%E3%80%8B%7C%E2%98%86%E8%89%B2%E3%82%AC%E3%83%BC%E3%83%AB%E3%83%89%E3%83%AD%E3%83%83%E3%83%97}}

ssastry moved this task from Backlog to Read Views on the Parsoid board.Mar 15 2018, 8:56 PM
Pchelolo added a subscriber: Pchelolo.

We're getting closer to launching language variants support in RESTBase, so I think we need to start the discussion about how page previews should be using the new API. Technically it's not hard, just add accept-language header to the summary request, but I guess it should be a product decision on which exact variant to request - the user's preferable, the one a current page is in now or other?

phuedx added a subscriber: phuedx.Jun 28 2018, 4:09 PM

Per discussion on the Audiences-Services sync meeting, the initial step on the Page-Previews side could be to just explicitly set the Accept-Language header to the current wiki language code with no variant. This will prevent Varnish cache splitting for different browser/OS language settings and will unblock Services with deploying the summaries support of variants into production.

Afterward, as the product decision has been made on what exact variant to request for each situation, Reading web will be free to enable and test variants per-wiki in their own pace.

What's the plan for extracting the language code from the language inside endpoints? I see this middleware that looks interesting.
I'm using https://www.npmjs.com/package/accept-language

Note, we may have some troubles serving languages where the mediawiki codes do not match their standardise equivalents e.g. 'zh-classical', 'zh-min-nan',

See: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/mobileapps/+/434193 for an example usage.

Per discussion on the Audiences-Services sync meeting, the initial step on the Page-Previews side could be to just explicitly set the Accept-Language header to the current wiki language code with no variant. <snip />

Broken out into T198619: Send the Accept-Language header with RESTBase API request for page preview requests.

What's the plan for extracting the language code from the language inside endpoints? I see this middleware that looks interesting.
I'm using https://www.npmjs.com/package/accept-language
Note, we may have some troubles serving languages where the mediawiki codes do not match their standardise equivalents e.g. 'zh-classical', 'zh-min-nan',

Yeah, I rejected the accept-language package for that reason in the work on https://gerrit.wikimedia.org/r/c/mediawiki/services/mobileapps/+/439997 but that's currently stalled.

@cscott's work on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/443687 will allow MCS to pass either the industry standard or MediaWiki codes to the MW API and it "Just Works™". I'd expected to generalise whatever he writes into some JS for server-side code so that we can do the same with Parsoid, and then gently migrate all clients over to the industry standard and hide the MW weirdness as much as possible.

See: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/mobileapps/+/434193 for an example usage.

RHo added a subscriber: RHo.Jul 25 2018, 4:09 PM
Pchelolo closed this task as Resolved.Jul 30 2018, 8:06 PM
Pchelolo claimed this task.

Everything is done here on the API side and on the popups side. For example, on sr.wikipedia.org the content of the preview respects user's variant settings.

However, Parsoid currently supports conversions only on a subset of wikis, so popups will get variants support as Parsoid adds more conversions.

Nothing really needs to be done here anymore, so I think we can close the ticket.

🎉🎉🎉