Thanks for the correction. I didn't realise this was a different extension.

I tested it out on https://translate.wmcloud.org/ and unfortunately it seems like it can't handle anything longer. The output is a disappointing mix of zh_Hans and zh_Hant. I see that in https://gerrit.wikimedia.org/r/c/mediawiki/services/machinetranslation/+/991469 zh-yue is incorrectly mapped to zh_Hant, which I would place the blame on; but still, its inability to even output zh_Hant properly is disheartening.

Please do not consider yue (Cantonese) support at this time. Like NLLB/FLORES-200 this dataset again suffers from the common problem of "yue" data not being yue. To quote the paper itself, "pretty low quality; mostly not Canto". The evaluation scores alone are painfully low, and even worse then NLLB if I'm interpreting them correctly.

In T304865#9136870, @Pginer-WMF wrote:

We can make more efforts to reach to more people, explain changes better and involve more people of the community, but I don't consider our approach "incredibly disrespectful". We are approaching this announcement with a very honest way and willing to adjust or revert any decisions based on the community input.

I have no doubts your team is approaching the work here in good will, but the communication has proven ineffective, hence my previous comment, forwarding the local sentiment in hopes to push for more inclusive dialogue going forward.

A user left a new comment on the village pump in response to @UOzurumba's comment, and I'd like to bring it up here so it doesn't get lost when archived. It's honestly audacious and incredibly disrespectful to treat the lack of response as community consensus to allow this to be pushed forward, especially when the original announcement was vague and ambiguous, and that the entire notice was never translated to Cantonese, denying users who don't know English from voicing their concerns. Sincerely I would like to ask everyone involved in that decision to be mindful of this when introducing and communicating any further changes.

I believe this new status best reflects the total lack of community support and the blocking issues with LanguageConverter that prevent this feature from being useful or beneficial.

@Pginer-WMF I noticed this diff article: https://diff.wikimedia.org/2023/06/13/mint-supporting-underserved-languages-with-open-machine-translation/, and I think it's worth reiterating that there is no such thing as the "yue-Shumianyu" "variant". "Shumianyu" is a cmn label for written Mandarin (cmn/zho), and cmn and yue are mutually unintelligible. Again I have been unable to locate any information about the NLLB training data for yue, so unfortunately I cannot comment directly on its validity; but seeing that the "yue" dataset for FLORES is entirely not yue , it would not surprise me that the NLLB data is equally invalid and in a completely different, mutually unintelligible language.

Can we just acknowledge that this current implementation of the yue LanguageConverter is a bad idea? This was pushed forward despite many existing issues with LanguageConverter and community opposition, while bringing negligible benefit to the project. (I'd even argue that the issues outweigh the benefits.) There is little demand for the feature to begin with, that you'd even have to dig up a 10 year old thread on a defunct discussion page to justify this, and as far as I am aware the existing JavaScript gadget already covers most use cases. Also, to some extent it's more useful than LC, since it does not offer fragmented control over the interface and content script variants seperately as LC does.

Another issue that's been bugging me is that bare urls in messages are now shown in this LanguageConverter syntax: -{R|https://example.org/}- (which apparently means "raw content"), even though it's not present in the message sources. I can't find any further info how to get rid of it.

I do want to note that I was able to consistently reproduce this issue on the first two or three days after the changes went live. It occurs less frequently now.

In T59106#8898101, @Winston_Sung wrote:

In T59106#8898093, @H78c67c wrote:

So, right now there are two issues:

1. Some messages loaded after the initial page load fall back to English

Is there any examples? I never encountered this issue. If there's any, we should encounter them for zh-tw too (as zh-tw messages stored under zh-hant ).

So, right now there are two issues:

1. Some messages loaded after the initial page load fall back to English
2. All customised messages are being disregarded by default

In T59106#8887575, @Func wrote:

It turned out the culprit is the user option manager:

	private function loadOriginalOptions(
		UserIdentity $user,
		int $queryFlags = self::READ_NORMAL,
		array $data = null
	): array {
		$userKey = $this->getCacheKey( $user );
		$defaultOptions = $this->defaultOptionsLookup->getDefaultOptions();
		$isTempUser = $this->userNameUtils->isTemp( $user->getName() );
		if ( !$user->isRegistered() || $isTempUser ) {
			// For unlogged-in users, load language/variant options from request.
			// There's no need to do it for logged-in users: they can set preferences,
			// and handling of page content is done by $pageLang->getPreferredVariant() and such,
			// so don't override user's choice (especially when the user chooses site default).
			$variant = $this->languageConverterFactory->getLanguageConverter()->getDefaultVariant();
			$defaultOptions['variant'] = $variant;
			$defaultOptions['language'] = $variant;
			$this->originalOptionsCache[$userKey] = $defaultOptions;
			return $defaultOptions;
		}

But this should only happen for logged-out users, @H78c67c could you confirm?

In T59106#8885606, @H78c67c wrote:

Is there any timeline for when this mess will be fixed? DiscussionTools and MobileFrontend are both failing to serve some of the messages translated to Cantonese. This is unacceptable.

The default UI language is now yue-hant instead of yue, and the default yue messages are being shown instead of the locally customized messages.

In T336019#8858802, @Diskdance wrote:

jquery.tipsy is used by a default gadget on Chinese Wikipedia providing cross-language link support, which has wide usage in the wiki.

Is there any timeline for when this mess will be fixed? DiscussionTools and MobileFrontend are both failing to serve some of the messages translated to Cantonese. This is unacceptable.

In T59106#8884379, @Winston_Sung wrote:

In T59106#8884359, @Winston_Sung wrote:

Might need to copy them into language subpages.

I'd say this is actually a side effect which should be fixed.

In T59106#8884359, @Winston_Sung wrote:

In T59106#8884221, @H78c67c wrote:

Right now no customized interface messages are visible unless users deliberately set their UI language preference to yue

Might need to copy them into language subpages.

Doesn't this go against the goal of this task, that is implementing LC to the extent of reaching feature parity with the JS gadget (and no more)? I don't think anyone consented to this.

Also, DiscussionTools is now having trouble loading some of the UI messages and is defaulting back to English.

Right now no customized interface messages are visible unless users deliberately set their UI language preference to yue

This seems to have changed the default UI language to be something other than yue (yue-hant presumably). By default, all interface messages are now shown as their MediaWiki defaults.

Thanks for the clarification. Assuming the search behaviour remains the same (no normalization to Hans), personally I'm fine with this change. We have an AbuseFilter in place to block all LC syntax so those side effects can be ignored.

Given the above opposing comments and the various concerns mentioned in https://zh-yue.wikipedia.org/wiki/Wikipedia_talk:用字轉換, I believe it's best to hold off enabling the change on Wikipedia at the very least. Granted, some of the issues are from as early as 2006, and LC has evolved in the meantime; but surely it doesn't hurt to have a local discussion to gauge the community's attitude towards LC in 2023 first.

While this is certainly interesting, as the Published Translations page already acknowledges,

The Opus project does look interesting. Thank you for the idea! I am curious how some poorly translated MT sentences and old (bad) revisions of translated sentences managed to get into their Wikimedia dataset, and I'd love to help provide some input to improve yue corpora.

I want to note that for individual, commonly used phrases, Bing does generate accurate responses more often, e.g.

I apologise for a mistake I made above. I misread "Yue-Kouyu" as "Yue-Kuoyu". The so-called "Kouyu" (口語; Mandarin Chinese word for "Spoken Language") would indeed be closer to what's being used in yue.wikipedia.org. However, Bing does not seem to generate "Kouyu", but is slightly better than NLLB-200 since it does generate commas properly and does sprinkle some uniquely Yue words here and there. Otherwise it generates mostly cmn content.

• Yue "Variants": This sounds like the exact same issue @Al12si pointed out above. It sounds like Yue-Kuoyu (國語; Mandarin Chinese word for "Mandarin") and Yue-Shumianyu (書面語; Mandarin Chinese word for "Written Language") are really just Mandarin Chinese (cmn). Editors on yue.wikipedia.org contributing in these are generally invited to contribute to zh.wikipedia.org instead, since that's where all the Mandarin/guoyu/"shumianyu" Wikipedia content are hosted. cmn is mutually unintelligible with yue.
• Defaults: NLLB-200 isn't even generating commas properly, isn't in the right language and only generates plain text. I honestly see little value in keeping it at all. Instead of dealing with the useless stuff it generates it's probably easier to just start from scratch.

I appreciate WMF's intention to make translating more accessible, but defaulting to this poor quality MT service to newcomers is absolutely unacceptable. It's true that virtually any tools can be misused, but we shouldn't be offering this awful experience to new users to begin with. Can we limit this to experienced users, or at the very least make it opt-in instead of opt-out?

I apologise for hijacking this thread, but since we're on the topic of MT in CX and since I'm not sure where's a better venue to report this, I'd like to point out that the FLORES-200 dataset for yue_Hant is, in fact, completely not yue_Hant. I am not entirely sure what NLLB-200 was trained on, but I downloaded the FLORES dataset used for benchmarking it and of all the 2009 lines in yue_Hant.dev and yue_Hant.devtest, not a single line is in yue (0/2009; 0.00%; zero). This explains the extreme low quality output, but not the half-width commas.

This is probably out of scope of this particular thread, but seeing the above I've decided to revisit CX, now with machine translation, and all I can say is I'm shocked by how low the MT quality is.
Using NLLB-200:
Original:

The Hastings line is a secondary railway line in Kent and East Sussex, England, linking Hastings with the main town of Tunbridge Wells, and London via Tonbridge and Sevenoaks. Although primarily carrying passengers, the railway also serves a gypsum mine which is a source of freight traffic. SE Trains operates passenger trains on the line, and it is one of their busiest lines.
...
The Hastings line is built over the difficult, forested, and hilly terrain across the High Weald and sandstone Hastings Beds, necessitating the construction of eight tunnels between Tonbridge and the south coast seaside resort of Hastings. The SER was anxious to construct the line as economically as possible, since it was in competition with the LBSC to obtain entry into Hastings and was not in a strong financial position in the mid-1840s.

Translated:

黑斯廷斯線係英格蘭肯特郡同東蘇塞克斯郡嘅一條二級鐵路線,連接黑斯廷ছ同主要城鎮坦布里奇威爾斯,同埋通過東布里奇同塞文奧克斯連接倫敦。 儘管主要運載乘客,鐵路亦都為石膏礦提供服務,石膏礦係貨運交通嘅源頭。 SE 列車喺線上運營客運列車,係佢哋最繁忙嘅線路之一。
...
黑斯廷斯線建在高威爾德和黑斯廷士沙石床上,難以建造,森林和山地地,需要在東布里奇和南海岸海灘度假村黑斯廷茲之間建造八條 tunnels。 SER急於在經濟上盡可能建設這條線路,因為它與LBSC競爭,以獲得黑斯廷斯的入境,並且在1840年代中期並沒有強大的財務地位。

For anyone interested here are 2 queries to filter the translated articles in 2022:

• zh -> yue: https://quarry.wmcloud.org/query/71596
• yue -> zh: https://quarry.wmcloud.org/query/71598

On top of that, personally I don't think it is a good idea to enable it by default for all registered users. CX has its merits, and some editors have been using it to create numerous high quality articles. But with CX's almost intruding ubiquity, this change makes it too accessible, even to those who are unfamiliar with the local policies and style guides, putting unnecessary burden on the already struggling editor community.

The main issue raised in the thread I linked above, and that many people find problematic, is the gray links entry point to CX, on the left sidebar in the language list, causing confused users to follow a link to a special page encouraging them to translate the page to a different language.

In T304865#8629365, @KartikMistry wrote:

In T304865#8627370, @H78c67c wrote:

This unexpected change is attracting some negative attention and reaction at https://zh-yue.wikipedia.org/wiki/Wikipedia:城市論壇_(技術)#點解加咗個鼓勵人翻譯到中文維基嘅功能？

Please see: https://zh-yue.wikipedia.org/wiki/Wikipedia:%E5%9F%8E%E5%B8%82%E8%AB%96%E5%A3%87_(%E6%8A%80%E8%A1%93)/%E6%AD%B8%E6%AA%94/2022%E5%B9%B46%E6%9C%88#Enabling_Content_and_Section_translation_tool_in_Cantonese_Wikipedia

This unexpected change is attracting some negative attention and reaction at https://zh-yue.wikipedia.org/wiki/Wikipedia:城市論壇_(技術)#點解加咗個鼓勵人翻譯到中文維基嘅功能？

Yup. It has been fixed now.

Note: a substantial amount of bad edits have been revdel'd on zh_yuewiki. Would that be an issue?

The change to yue should be reverted ASAP, having caused numerous undesirable side effects as brought up in https://zh-yue.wikipedia.org/wiki/Wikipedia:城市論壇_(技術)#MediaWiki_介面訊息今後無粵文翻譯就會預設用zh-hk或zh-hant顯示

I would like to state that I cannot think of any justifiable reason for this change. Can you explain why you changed the fallback chain to what it is please?

Is there any reason why the fallback chain for yue is modified? The change has caused confusion, concern and shock in yue.wikipedia.

New opposing comments emerged, changing status to stalled.

Any plans for the patch to be merged?

Wow, this is really fast. Thank you.

Duplicate of T219733.

Still not fixed on version 6.7.1.

A user in yuewiki suggested another way to do this. (https://zh-yue.wikipedia.org/wiki/Wikipedia:城市論壇_(提議)#Yue.wikipedia.org) Is it possible to create yue.wikipedia.org and import all pages from zh-yue?

There is another issue which I am not sure if it is related, but the Wiki Loves Folklore CentralNotice is in Urdu when the interface language is English.

Maybe related to the super annoying T30441 ? If that's the case, then:
Task 30441 and this task in a nutshell: Cantonese (yue) is treated as a variant of Chinese (zh) because of it's db identifier/subdomain zh-yue

So seems like this bug describes the same issue as in T219733. Could anyone help close/merge this task?

Working on latest build. Thanks!

Problem solved for the Top Read ones, but no articles except the Top Read ones could be edited now.

Any progress on this?

@Urbanecm In line 7074 it should be 103 instead of 104, causing bad title when visiting https://yue.wiktionary.org/wiki/Appendix talk:test

In T205546#4793440, @Urbanecm wrote:

In T205546#4791810, @Hello903hello wrote:

The additional namespaces haven't been added to InitialiseSettings.php.

Do you want English aliases?

Sure

The additional namespaces haven't been added to InitialiseSettings.php.

In T209820#4770669, @Ladsgroup wrote:

...and language of zh-yue wiktionary is "yue" and it's already there.

I'm confused. There seems to be no "yue" throughout the whole InterwikiSortOrders.php.

Does "yue" need to be added to InterwikiSortOrders.php? Seems like there's only zh-yue now.

In T209693#4757863, @ArielGlenn wrote:

So why does yue.wikipedia go to zh-yue.wikipedia?

T10217 and T30441
Initially created at zh-yue in 2006, Cantonese Wikipedia didn't have a proper ISO 639-3 code. But soon after that, the code "yue" got approved. However, wiki renaming has been impossible (possible, but problems would arise).

WIkidata support seems to be missing, as reported in the Village Pump.

We just need to add proper redirect rules for zh-yue.wiktionary.org to yue.wiktionary.org at the current stage, period.

Probably this is not the right place to request interwiki prefix changes, but currently most wikimedia sites' Cantonese Wiktionary link is pointing to the domain zh-yue.wiktionary.org instead of yue.wiktionary.org. Also, zh-yue.wiktionary.org/wiki/$1 doesn't redirect to yue.wiktionary.org/wiki/$1. Hope this would be fixed since this has caused some inconvenience.

Thank you for explanation. To be honest, I really do think the "talk" suffix should be the same for all namespaces. I don't find intuitive having following namespaces:

• NS_TALK: 傾偈
• NS_TEMPLATE_TALK: 模傾偈

but having 維基辭典討論.

I guess that's why the Cantonese Wikipedia uses English namespaces.
Anyway, 維基辭典傾偈 is fine, too.

@Urbanecm : 傾偈, as in $1傾偈 , literally means chatting, while 討論 means discussion (they're more or less the same though). Since there's no community consensus (no response), I would go for 維基辭典討論 (Wiktionary talk).

@Urbanecm : I noticed a typo at line 3108 in the InitaliseSettings.php. It should be yuewiktionary instead of yoewiktionary.