Page MenuHomePhabricator
Create Task
Maniphest T327379

Update mobileapps to use new Language variant codes
Closed, ResolvedPublic
Actions

Assigned To
vadim-kovalenko
Authored By
Jgiannelos
Jan 19 2023, 8:41 AM
Tags
Referenced Files
F37107510: zh-hans
Jun 16 2023, 10:49 PM
F37107508: zh-tw
Jun 16 2023, 10:49 PM
F37107512: sr-ec
Jun 16 2023, 10:49 PM
F37107515: sr-el
Jun 16 2023, 10:49 PM
Subscribers
Aklapper
cscott
Dbrant
Jgiannelos
MSantos
Tsevener
vadim-kovalenko

Description

There is an issue found in the restbase test suite after some test failures:
It looks like language variants handling doesn't work properly on wikitext to mobile-html conversion.

This is the patch with the failing tests:
https://github.com/wikimedia/restbase/pull/1314

Details

SubjectRepoBranchLines +/-
Message::inLanguage(): accept Bcp47Code as well as string and Languagemediawiki/coremaster+31 -8
Mobileapps: Update mobileapps to use new Language variant codesmediawiki/services/mobileappsmaster+86 -88
Use Bcp47Code when interfacing with Parsoidmediawiki/coremaster+748 -412
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jgiannelos created this task.Jan 19 2023, 8:41 AM
Restricted Application added a project: Product-Infrastructure-Team-Backlog-Deprecated. · View Herald TranscriptJan 19 2023, 8:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Jgiannelos removed a project: Product-Infrastructure-Team-Backlog-Deprecated.Jan 19 2023, 8:42 AM
Jgiannelos added subscribers: vadim-kovalenko, MSantos.
Aklapper renamed this task from Languave variant handling in mobileapps is inconsistent with mw core variants to Language variant handling in mobileapps is inconsistent with mw core variants.Jan 19 2023, 9:07 AM
• JMcLeod_WMF removed a project: Content-Transform-Team.Jan 19 2023, 3:04 PM
Jgiannelos claimed this task.Jan 27 2023, 11:00 AM
Jgiannelos moved this task from Backlog to In Progress on the Content-Transform-Team-WIP board.
cscott subscribed.Jan 30 2023, 4:20 PM

Two related patches, one in parsoid: https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/884358 (merged in dbbd78938c878f3a799c4ee2a89c337ac0062677 and riding the train this week) and one in core (https://gerrit.wikimedia.org/r/c/mediawiki/core/+/853380, still in progress.)

gerritbot added a comment.Jan 30 2023, 4:20 PM

Change 853380 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] WIP: Use Bcp47Code when interfacing with Parsoid

https://gerrit.wikimedia.org/r/853380

gerritbot added a project: Patch-For-Review.Jan 30 2023, 4:20 PM
Jgiannelos removed Jgiannelos as the assignee of this task.Feb 10 2023, 1:33 PM
Jgiannelos moved this task from In Progress to Backlog on the Content-Transform-Team-WIP board.
cscott added a subtask: T323052: Parsoid returns mediawiki-internal codes in the Content-Language header.Feb 23 2023, 3:27 PM
gerritbot added a comment.Mar 13 2023, 7:11 PM

Change 853380 merged by jenkins-bot:

[mediawiki/core@master] Use Bcp47Code when interfacing with Parsoid

https://gerrit.wikimedia.org/r/853380

cscott renamed this task from Language variant handling in mobileapps is inconsistent with mw core variants to Update mobileapps to use new Language variant codes.Apr 17 2023, 3:44 PM
vadim-kovalenko added a comment.Apr 19 2023, 1:46 PM

@Jgiannelos , @cscott considering https://phabricator.wikimedia.org/T117845#8576044 - does this make a blocker for me to replace sr-el / sr-cr variant codes to sr-Latn / sr-Cyrl in mobileapps service?

cscott added a comment.May 9 2023, 2:13 PM

The external interfaces (HTTP headers) already support sr-Latn and sr-Cyrl so T117845 is not a blocker -- T117845 is only for migrating the internal mediawiki codes.

vadim-kovalenko claimed this task.May 11 2023, 8:46 AM
vadim-kovalenko moved this task from Backlog to In Progress on the Content-Transform-Team-WIP board.
gerritbot added a comment.May 11 2023, 8:51 AM

Change 918550 had a related patch set uploaded (by Vadim Kovalenko; author: Vadim Kovalenko):

[mediawiki/services/mobileapps@master] Mobileapps: Update mobileapps to use new Language variant codes

https://gerrit.wikimedia.org/r/918550

vadim-kovalenko moved this task from In Progress to Code Review on the Content-Transform-Team-WIP board.May 11 2023, 9:02 AM
vadim-kovalenko moved this task from Code Review to In Progress on the Content-Transform-Team-WIP board.May 11 2023, 9:40 AM
vadim-kovalenko added a comment.May 12 2023, 12:10 PM

I updated the language codes listed in lib/wikiLanguageMapping.json. As a guideline, I used ISO 15924 and ISO_3166-1.

  1. ISO_15924 - Subtags whose 'Type' field is 'script' (in other words, subtags defined by ISO 15924) MUST use titlecase. Example: Cyrl, Latn, Arab.
  2. ISO_3166-1 - Subtags whose 'Type' field is 'region' (in other words, the non-numeric region subtags defined by ISO 3166-1) MUST use all uppercase. Example: BR (for Brasil), CA (for Canada).

Check RFC5646#2.1.1 and A RFC5646#3.1.4 for more details.

vadim-kovalenko moved this task from In Progress to Code Review on the Content-Transform-Team-WIP board.May 12 2023, 12:11 PM
MSantos moved this task from Code Review to To Deploy on the Content-Transform-Team-WIP board.May 22 2023, 3:13 PM
MSantos moved this task from To Deploy to Code Review on the Content-Transform-Team-WIP board.
vadim-kovalenko moved this task from Code Review to In Progress on the Content-Transform-Team-WIP board.May 26 2023, 7:29 AM
Tsevener mentioned this in T338079: Update iOS hardcoded language variant codes to BCP 47.Jun 3 2023, 1:04 AM
gerritbot added a comment.Jun 14 2023, 10:42 AM

Change 918550 merged by jenkins-bot:

[mediawiki/services/mobileapps@master] Mobileapps: Update mobileapps to use new Language variant codes

https://gerrit.wikimedia.org/r/918550

vadim-kovalenko moved this task from In Progress to To Deploy on the Content-Transform-Team-WIP board.Jun 14 2023, 11:07 AM
Seddon added projects: Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog.Jun 14 2023, 2:12 PM
Tsevener subscribed.Jun 16 2023, 10:49 PM

Update from a quick test in simulator in the beta cluster. Looks to me like backwards compatibility is working.

https://sr.wikipedia.beta.wmflabs.org/api/rest_v1/page/mobile-html/BigPage

Visited this page on SR Wikipedia, sending sr-el in the Accept-Language header, and it correctly displays Latin characters. Sending sr-ec in header correctly displays Cyrillic characters.

sr-el

sr-el (1×575 px, 332 KB)

sr-ec

sr-ec (1×575 px, 328 KB)

https://zh.wikipedia.beta.wmflabs.org/api/rest_v1/page/mobile-html/%E6%B8%AF%E9%90%B5%E5%B8%82%E5%8D%80%E7%B6%AB%E9%9F%93%E8%A3%BD%E5%88%97%E8%BB%8A

Went to this page with both zh-tw and zh-hans in the Accept-Language header. A screenshot of both article views indicates the characters are changing in response.

zh-tw

zh-tw (1×575 px, 305 KB)

zh-hans

zh-hans (1×575 px, 302 KB)

MSantos added a parent task: T336146: [EPIC] FY 23-24 unplanned and maintenance for Content Transform Team.Jun 19 2023, 11:19 AM
MSantos added a project: Essential-Work.Jun 23 2023, 1:26 PM
Dbrant subscribed.Jun 26 2023, 4:36 PM

Confirmed that it's working as expected for Android, too.

Dbrant moved this task from Needs Triage to Tracking on the Wikipedia-Android-App-Backlog board.Jun 27 2023, 1:43 PM
LGoto moved this task from Needs Triage to Tracking on the Wikipedia-iOS-App-Backlog board.Jun 27 2023, 6:20 PM
Jgiannelos moved this task from To Deploy to To Verify on the Content-Transform-Team-WIP board.Jul 4 2023, 2:05 PM
MSantos moved this task from To Verify to To Deploy on the Content-Transform-Team-WIP board.Jul 4 2023, 2:05 PM
MSantos moved this task from To Deploy to To Verify on the Content-Transform-Team-WIP board.
MSantos closed this task as Resolved.Sep 12 2023, 1:49 PM
vadim-kovalenko mentioned this in T278272: Rest API endpoints do not respect Accept-Language header for some Wikis.Sep 21 2023, 6:04 PM
cooltey mentioned this in T343387: /page/summary/ and /page/mobile-html/ do not get the latest article description in zhwiki.Apr 6 2024, 12:21 AM
gerritbot added a comment.Apr 6 2024, 1:54 AM

Change #1017420 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] Message::inLanguage(): accept Bcp47Code as well as string and Language

https://gerrit.wikimedia.org/r/1017420

gerritbot added a comment.Fri, Apr 12, 3:40 PM

Change #1017420 merged by jenkins-bot:

[mediawiki/core@master] Message::inLanguage(): accept Bcp47Code as well as string and Language

https://gerrit.wikimedia.org/r/1017420

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.1; 2024-04-16).Fri, Apr 12, 4:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL