I'm not sure if language updater bot will add them back even if I delete them in a patch if these messages still exist on translatewiki

Which system is meant in "have not been removed from the system"?

In T229992#5398803, @Aklapper wrote:

Which system is meant in "have not been removed from the system"?

I initially thought these messages still exist in the software, then I realize the software is written in a way where if you set a fallback, these message will still psudo-exist, even if they are not defined in that specific variation. So if SYS:MSG/zh-sg do not exist, but fallback is set to SYS:MSG/zh-hans, it seems the system still interprets it as if the message exist.
So it appears the fallback system is written in two place, one happens at reading MSG, one happens while reading default, which pseudo-create non-existent MSG/zh-sg using its fallback MSG/zh-hans.

This may also have caused T228876#5392631

@RazeSoldier: It's up to teams what they plan to have on their workboard, so I'm removing Platform Engineering again.

Almost all interface messages are obtained via Message::fetchMessage(), it get real message from MessageCache::get().

The logical for MessageCache::getMessageForLang() (called by ::get()) display, first try to get the message from MediaWiki namespace, if it fails, check the CDB cache, and finally check the database for all of the fallback languages. This means that if the CDB cache exists, the fallback languages will never be used. But the logic of this method has not changed for a long time, I suspect the problem related to CDB cache.

Change 529350 had a related patch set uploaded (by 星耀晨曦; owner: 星耀晨曦):
[mediawiki/core@master] [DO NOT MERGE] Debug for T229992

https://gerrit.wikimedia.org/r/529350

I wrote a test to illustrate the situation. But the integration test results are very strange, the test fails under PHP and the test passes under HHVM.

In T229992#5405026, @RazeSoldier wrote:

I wrote a test to illustrate the situation. But the integration test results are very strange, the test fails under PHP and the test passes under HHVM.

According to my assumption, the CI test of the patch should fail. But the job under HHVM did not fail. I don't know what is causing this difference. But given that we are about to drop HHVM, we don't have to care about this difference. (although this difference is likely to be the cause for T228876)

From the description:

After further digging, it seems the problem exist because the read order seems to be MW:MSG/zh-sg, SYS:MSG/zh-sg, SYS:MSG/zh-hans, MW:MSG/zh-hans.
I would propose something like MW:MSG/zh-sg, SYS:MSG/zh-sg; MW:MSG/zh-hans; SYS:MSG/zh-hans. where SYS denotes default in config file, and MW denotes messages created locally in db.

I think Chinese has some special casing, where /zh alone isn't used and an explicit variant must be used instead. However, there are other desired behaviors as explained in T217357: MediaWiki:Talkpageheader message does not handle language fallbacks for other languages like English.

If we use English as an example, the desired behavior would probably be: MW:MSG/en-ca, MW:MSG/en, SYS:MSG/en-ca, SYS:MSG/en

Basically, if you customize a non-variant language, it should take precedence over all variants of that language, unless the variant is also personalized. Otherwise, wiki administrators would need to personalize all current and future variants of a language when editing a message. Using this fallback strategy should generally get better results than returning the unmodified one when a user has set a language variant in preferences other than the wiki's default language. This will cause that user to see the wiki without any personalization in interface messages.

This wasn't a problem before, but recent MediaWiki versions come with more language variants and it causes this problem to surface more often.

Note that, in cases where the language is also the wiki's default language MW:MSG/lang should be simply MW:MSG