Page MenuHomePhabricator

Need variant options for Chinese in Language Settings
Closed, ResolvedPublic

Assigned To
Authored By
digdog
May 11 2016, 7:55 AM
Referenced Files
F4559945: Simulator Screen Shot Oct 5, 2016, 10.41.54 AM.png
Oct 5 2016, 2:51 PM
F4557220: IMG_0011.PNG
Oct 4 2016, 9:56 PM
F4530677: Simulator Screen Shot Sep 27, 2016, 4.23.18 PM.png
Sep 27 2016, 8:29 PM
F4530693: Simulator Screen Shot Sep 27, 2016, 4.27.28 PM.png
Sep 27 2016, 8:29 PM
F4527369: Screenshot (26 Sep 2016 22:19:03)
Sep 26 2016, 9:07 PM
F4527127: hans.png
Sep 26 2016, 7:16 PM
F4527126: hant.png
Sep 26 2016, 7:16 PM
F4526920: Simulator Screen Shot Sep 26, 2016, 1.47.27 PM.png
Sep 26 2016, 5:55 PM
Tokens
"Like" token, awarded by debt."100" token, awarded by Liuxinyu970226.

Description

As a Trad. Chinese user, it's annoying to see Simp. Chinese contents mixed with Trad. Chinese contents when using Chinese as app's primary language. It would be really nice if app can behave like web that provided variant options for Chinese, e.g. zh-tw, that will only show specific variant contents.

This is a much more important feature than having "Classic Chinese" option in Language Settings.

Steps to reproduce

  1. Launch iOS app
  2. Use Chinese as primary language in Language Settings

Expected results

After choose Chinese, it should show variants like "Trad. Chinese - Taiwan" (zh-tw) for user to further specify the content they want to see.

And if user choose "Trad. Chinese - Taiwan", content should show up in Trad. Chinese.

Actual results

After choose Chinese, there's no variant options.

And the content that app shows are mixing with Simp. Chinese and Trad. Chinese, which is very annoying.

Event Timeline

This might be a bit complex. My understanding is this is done via some kind of transform applied to the content. Will require research on how to best support this in the app.

Per the entry on Chinese Wikipedia, from circa Jan 2005, using "zh-tw" and "zh-cn" to distinguish between the two forms of writing was no longer supported with the introduction of:
"...a mechanism to automatically convert different characters and vocabulary items into the user's local ones, according to the user's preference settings, which may be set to one of two settings that convert the script only, or one of five settings that also take into account regional vocabulary differences" [1]

This is something we can look at doing in the app, i.e., when a user with Chinese Simplified as their iOS language that zh Wikipedia displays articles in Simplified Chinese; and when the iOS language is set to Chinese Traditional the same articles in traditional characters.
An example article used is the article for Hair - written in Traditional characters as "頭髮", and in Simplified as "头发".

Steps to reproduce

  1. Set iOS language settings to "Chinese, Simplified" (简体中文).
  2. Open the Wikipedia app with "中文" zh wikipedia also as the main language.
  3. Search for "头发" (hair).

Expected
Article result should show in Simplified Chinese characters.

Actual
Article result shows in Traditional characters (the original way the article was created)

iOS app (no transform)
Simulator Screen Shot 9 Aug 2016, 2.22.56 PM.png (1×750 px, 516 KB)
On Safari Browser (transform applied)
Simulator Screen Shot 9 Aug 2016, 2.30.58 PM.png (1×750 px, 393 KB)

[1] https://en.wikipedia.org/wiki/Chinese_Wikipedia#Automatic_conversion_between_traditional_and_simplified_Chinese_characters

An alternate solution would be to do what the Android app does, and separate the single "中文" Chinese (zh) language option we currently have on iOS into two options: (a) "繁體中文" Traditional Chinese, and (b) "简体中文" Simplified Chinese and apply the transform depending on the language selected.

@RHo that sounds like a good solution to get this working soon.

According to Piwik ~6% of visits are from devices with Locale set to Chinese, and according to iTC actual sessions from China itself are roughly 1% a day, our 15th most frequent "territory" (in iTunes jargon).

JMinor raised the priority of this task from Medium to High.Sep 19 2016, 6:20 PM

@RHo @JMinor

It's easier to implement the desired behavior from Rita's original steps (show traditional or simple based on system prefs) rather than show traditional and simplified in the list. That change is ready to go here: https://github.com/wikimedia/wikipedia-ios/pull/922. Let me know if the actual preferred behavior is to show both languages in the list and I can implement that instead.

Per our meeting, this solution should be good to go, at least for beta testing.

Please file a followup bug for the need to manual refresh. Not a must fix, but good to track as a known issue.

I'm gonna ping Zhou from legal who is fluent in Chinese to see if he can be a guinea pig or maybe translate user feedback on this.

The search results are still showing the original character set the Chinese article was written in, and not applying the transforms.

Steps to reproduce:

  1. Change the iOS settings language to Simplified Chinese.
  2. Open the app and enter "头发" (hair in Simplified Chinese)

Expected:
Search result should should show the article for hair as "头发"

Actual:

Search result shows hair written in Traditional Chinese "頭髮"

Search results still showing in trad chinese.png (1×750 px, 179 KB)
| Though tapping on the result shows the article with correct simplified Chinese
Hair article in Simplified.png (1×750 px, 583 KB)

Note: this is the search results and also results when tapping to see "Other languages" an article is in. The following image after tapping on the language icon from the English "hair loss" article shows the traditional character for hair even though the iOS language is Simplified.

Hair loss - Other languages available - showing trad chinese.png (1×750 px, 100 KB)

@RHo - I'm not sure there's a way we can get those results to display in simplified. The app is passing the correct language headers to the search request should they ever support it. I verified that the mobile website has the same issue.

Simulator Screen Shot Sep 26, 2016, 1.47.27 PM.png (1×640 px, 146 KB)

Here's what I got for a different search:

Traditional:

hant.png (1×640 px, 153 KB)

Simplified:

hans.png (1×640 px, 153 KB)

Yes, looks like the transforms are not happening on the search results. Articles are just displaying in whatever variant the article was written in.

Seems like this needs to be fixed for search results across the board, looking for existing bug to link to, else can file a new ticket.

Also just tested and the same thing is occurring on the Android app as well.

Screenshot (26 Sep 2016 22:19:03) (1×720 px, 104 KB)

Testing Criteria:

  1. Open the "Settings" application
  2. Tap on "General"
  3. Tap on "Language & Region"
  4. Tap on "Other Languages..."
  5. Search for "Chinese, Simplified"
  6. Tap on the row with the subtitle "Chinese, Simplified"
  7. Tap on "Done"
  8. Tap on "Keep English" when the sheet appears
  9. Install and run Wikipedia
  10. Tap on the Search Icon in the top right
  11. Select "Chinese" as the search language
  12. Search for "Harvard"
  13. Tap on the row with "Redirected from: Harvard University"
  14. Verify that the writing in the article looks like this (pay attention to the last character of the title):

Simulator Screen Shot Sep 27, 2016, 4.23.18 PM.png (1×640 px, 352 KB)

  1. Uninstall Wikipedia
  2. Open the "Settings" application
  3. Tap on "General"
  4. Tap on "Language & Region"
  5. Tap on "Edit"
  6. Tap the Red minus icon next to "Chinese, Simplified" & confirm
  7. Tap on "Done"
  8. Tap on "Other Languages..."
  9. Search for "Chinese, Traditional"
  10. Tap on the row with the subtitle "Chinese, Traditional"
  11. Tap on "Done"
  12. Tap on "Keep English" when the sheet appears
  13. Install and run Wikipedia
  14. Tap on the Search Icon in the top right
  15. Select "Chinese" as the search language
  16. Search for "Harvard"
  17. Tap on the row with "Redirected from: Harvard University"
  18. Verify that the writing in the article looks like this (pay attention to the last character of the title):

Simulator Screen Shot Sep 27, 2016, 4.27.28 PM.png (1×640 px, 357 KB)

Is it possible that this problem is due to the fact that wmf mobile apps prefer to display target pages rather than redirects?

头发 is a redirect to 頭髮
When you use search box (top right on the desktop site) 头发 properly suggest 头发.
Clients that explicitly ask the API to resolve redirects will certainly display 頭髮 instead of 头发. (See www.wikipedia.org for example)

Could it be possible that the user doesn't have their phone OS settings set for just Traditional Chinese (rather than simplified) or that it is set to not change (transform) the text that a particular document is written in?

Sorry for late respond.

@JoeWalsh's #922 changes can't handle all the cases becasue iOS system langauge can be zh-HK, zh-TW or other combinations.

Please consider to use

+[NSBundle preferredLocalizationsFromArray:]

or

+[NSBundle preferredLocalizationsFromArray:forPreferences:]

Because NSBundle perfroms complex langauge matching under the hood so you don't have to™.

For example, if app is loading a Chinese wiki page: https://zh.wikipedia.org/wiki/苹果, you should use following code to load right variant based on current system language:

NSArray *wikiContentSupportedChineseVariants = @[@"zh-cn", @"zh-hk", @"zh-mo", @"zh-sg" @"zh-tw"];

NSArray *result = [NSBundle preferredLocalizationsFromArray: wikiContentSupportedChineseVariants];

If iOS system language is 'zh-hant-TW' or 'zh-TW', then you will get result @[@"zh-tw"] from wikiContentSupportedChineseVariants array. Then you can change wiki page URL to https://zh.wikipedia.org/zh-tw/苹果 to load content that matched iOS system language.

However, my original request was not based on iOS system language, becasue Chinese user may not use Chinese system language, they might use English or other system language. Also, based on the app's My Languages design, app should really follow the order of app's My Languages to display the content.

  1. Wiki app has all the supported Chinese variants (zh-cn, zh-hk, zh-mo, zh-sg, zh-tw) listed in My Language, and user can choose what they preferred, e.g. @"zh-sg", @"en", @"ja", @"zh-cn", @"zh-tw"]
  2. When user seach 'apple', you redirect to https://zh.wikipedia.org/wiki/苹果
  3. Call
NSArray *wikiContentSupportedChineseVariants = @[@"zh-cn", @"zh-hk", @"zh-mo", @"zh-sg" @"zh-tw"];
NSArray *myLanguages = @[@"zh-sg", @"en", @"ja", @"zh-cn", @"zh-tw"];

NSArray *result = [NSBundle preferredLocalizationsFromArray: wikiContentSupportedChineseVariants forPreferences: myLanguages];
  1. You will get @[@"zh-sg"], then you update URL to https://zh.wikipedia.org/zh-sg/苹果

REFERENCES
https://developer.apple.com/videos/play/wwdc2016/201/
https://developer.apple.com/library/content/technotes/tn2418/_index.html
https://developer.apple.com/reference/foundation/nsbundle/1417249-preferredlocalizationsfromarray?language=objc
https://developer.apple.com/reference/foundation/nsbundle/1409418-preferredlocalizationsfromarray?language=objc

Testing on iPhone 6s (iOS 10.0.1) with Wikipedia app 5.3.0 (956). Based on the results of recreating the testing steps @JoeWalsh provided, I've gotten similar results to the Harvard article in the screencap below. It looks fixed in this area. However since I don't speak or read Chinese I can't judge the correctness of the text.

IMG_0011.PNG (1×750 px, 445 KB)

@digdog

I've made a change that adds support for zh-cn, zh-hk, zh-mo, zh-sg, and zh-tw. Even if the system language is English, the order of preferred languages in system preferences is respected. For example, in this screenshot:

Simulator Screen Shot Oct 5, 2016, 10.41.54 AM.png (1×750 px, 121 KB)

The system language is english, but on Chinese wikipedia, zh-mo is used.

Leaving this open until we do the pubic beta, in case we need to make changes from user feedback.

@chelsyx (a native speaker of Mandarin) confirmed that the search results are also not being transformed on-wiki, e.g. a search on zhwiki for "China" displays a mix of Traditional and Simplified characters. Search does nothing special here in its handling of the characters, which likely means that the problem is tied up in the language converter in MediaWiki core. Special-casing a solution just for search seems unwise. Fortunately @EBernhardson has some expertise with this area of code, so he'll take a look at what's going on in core and see how difficult it would be to fix rigorously.

Some random findings:

If I log out (or use an incognito window) and set my accept-language to Chinese-Traditional, search results seem to be Traditional. If I switch to Chinese-Simplified, they are Simplified. If I set it to simply "Chinese", I get mixed results (probably in the format they've been stored in).

When I log in to zhwiki, my language preference is set to "zh", which seems to override my accept-language preference and give me mixed results.

It looks like the user's language pref is overriding the language-accept header. Maybe if one of those is a specialization of the other, we should use that, rather than just preferring WP prefs.

To be clear: all this is in Chrome... I'm still trying to get a handle on the desktop situation.

@digdog this is now available in our beta channel on TestFlight. Ping me if you need to be added.

Let us know if you see any issues with the implementation.

This will be released to the app store in a week or two.

I'm going to resolve this parent ticket as the core work is done. The remaining issue are either blocked on search API, or covered under T147342

@digdog and other subscribers, the version with improved character support is in the app store. See T147342 for follow-up issues specifically related to search suggestions/results.