Note that after logging in to your account, it will follow your preferences and the issue will be resolved. This particularly damages the user experience of anonymous readers. Both mobile and desktop versions have the problem.

https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/552549/2/wmf-config/InitialiseSettings.php deployed recently for T238803 removed:

'wgULSLanguageDetection' => [
	'default' => false,
	// see T203179
	'fixcopyrightwiki' => true,
],

Could this be the cause? (ping @Jdforrester-WMF)

There is another issue which I am not sure if it is related, but the Wiki Loves Folklore CentralNotice is in Urdu when the interface language is English.

It appears to be a problem with https://meta.wikimedia.org/w/index.php?title=CNBanner:Wikiloveslove2020-text/en-gb&oldid=19754638, not related to this issue.

In T246071#5915117, @Pcoombe wrote:

https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/552549/2/wmf-config/InitialiseSettings.php deployed recently for T238803 removed:

'wgULSLanguageDetection' => [
	'default' => false,
	// see T203179
	'fixcopyrightwiki' => true,
],

Could this be the cause? (ping @Jdforrester-WMF)

ULSLanguageDetection defaults to true in its extension.json, so that seems likely.

Change 574743 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Reinstate wgULSLanguageDetection setting

https://gerrit.wikimedia.org/r/574743

Is this causing cache pollution? If so, this should be merged ASAP to limit the damage of that.

Either caches are being polluted, or they are getting split (Vary: Accept-Language) or they are being bypassed.

Since sites are still up, likely not the last one. Since I am getting Finnish, probably not the first one either, so maybe cache splitting is happening?

Change 574743 merged by jenkins-bot:
[operations/mediawiki-config@master] Reinstate wgULSLanguageDetection setting

https://gerrit.wikimedia.org/r/574743

I just did some quick testing with curl. At a glance, it looks like varnish is respecting the vary: accept-language header and splitting cache.

Mentioned in SAL (#wikimedia-operations) [2020-02-25T14:15:52Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:574743|Reinstate wgULSLanguageDetection setting (T246071)]] (duration: 01m 03s)

Hm. The config change was deployed, and it seemed to do the right thing on mwdebug1001, but I’m still seeing English on https://zh-yue.wikipedia.org/wiki/%E9%A0%AD%E7%89%88.

In T246071#5915756, @Lucas_Werkmeister_WMDE wrote:

Hm. The config change was deployed, and it seemed to do the right thing on mwdebug1001, but I’m still seeing English on https://zh-yue.wikipedia.org/wiki/%E9%A0%AD%E7%89%88.

Works for me when logged out.

Yup, now it’s Chinese too. ruwiki (from duplicate task T246095) shows English UI for me at the moment, let’s see if that also fixes itself I guess?

Current status seems to be that the appservers behave correctly now (it may have taken them a bit to react to the change, it’s not clear); for a while, there seemed to be cached responses for some Accept-Language values in Varnish, which weren’t getting removed by their own, but now those appear to be gone as well, possibly having timed out. I can’t reproduce the issue on any page anymore.

It looks like we hit T236104: Cache of wmf-config/InitialiseSettings often 1 step behind. Looking at graphs things seems to be returning to normal, at least avg. response time and application server cluster load.

Started 00:53Z
Fix really applied: 14:35Z

Eurgh. Sorry about this, all. I assumed ULS's defaults were deploy-safe and didn't check.

Incident documentation is here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20200225-mediawiki_interface_language

As the issue seems to be fixed, shall we close this task? Or should it stay open for the User-notice or something?

In T246071#5916140, @Lucas_Werkmeister_WMDE wrote:

Incident documentation is here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20200225-mediawiki_interface_language

As the issue seems to be fixed, shall we close this task? Or should it stay open for the User-notice or something?

None of the affected users will ever read Tech/News, or have any way to converse with someone who does. I don't think it'd add value.

None of the affected users will ever read Tech/News, or have any way to converse with someone who does. I don't think it'd add value.

(I don’t necessarily disagree, I just saw that @Nikerabbit had added the tag.)

In T246071#5916155, @Jdforrester-WMF wrote:

In T246071#5916140, @Lucas_Werkmeister_WMDE wrote:

Incident documentation is here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20200225-mediawiki_interface_language

As the issue seems to be fixed, shall we close this task? Or should it stay open for the User-notice or something?

None of the affected users will ever read Tech/News, or have any way to converse with someone who does. I don't think it'd add value.

Some of them might (not everyone is logged in all the time), and there might be discussions on wikis about this behavior. I'd think it would be useful to let them know that this was an accident and not a planned changed, but I'll let the tech news writers decide.

(Either way – I have no strong opinion on including it or not – I don't think tickets need to stay open for Tech News.)

The post-mortem at https://wikitech.wikimedia.org/wiki/Incident_documentation/20200225-mediawiki_interface_language states

Reported by users at phabricator:T246071 and other tasks. There were no alerts as far as Lucas Werkmeister (WMDE) is aware.

and

Impact was not clear from the beginning, since the additional load was not significant to trigger any alerts (even through average app server response time jumped 33%)

and suggests

If human only, an actionable should probably be "add alerting".

Should a task for that be created?

Well, I have no idea what the alerting should be… adding an alert specifically for “Accept-Language is used” doesn’t sound reasonable to me.

If the load had increased enough to be a problem in itself, I assume there would have been alerts.