Page MenuHomePhabricator

Interface language using Accept-Language header value instead of $wgLanguageCode
Closed, ResolvedPublic

Description

Issue: Since a few hours ago the interface language for every wiki I visited became the first value of the Accept-Language header instead of their local languages.

How to reproduce:

  1. Sign out on Wikimedia projects
  2. Modify your Accept-Language header (change in browser settings, use an extension or launch an mitm attack on yourself)

In my case, my accept-language header was en-GB,en-US;q=0.9,en;q=0.8, so when I visited https://zh-yue.wikipedia.org/ (Cantonese Wikipedia), the interface language became English. When I changed the accept-language header to ja, the interface language became Japanese.

(This might have been described in T246043#5914612.)

Event Timeline

H78c67c created this task.Feb 25 2020, 8:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 25 2020, 8:41 AM
Pcoombe triaged this task as Unbreak Now! priority.Feb 25 2020, 10:04 AM
Pcoombe added a subscriber: Ahmad252.
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptFeb 25 2020, 10:05 AM
Ahmad252 added a comment.EditedFeb 25 2020, 10:14 AM

Note that after logging in to your account, it will follow your preferences and the issue will be resolved. This particularly damages the user experience of anonymous readers. Both mobile and desktop versions have the problem.

https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/552549/2/wmf-config/InitialiseSettings.php deployed recently for T238803 removed:

'wgULSLanguageDetection' => [
	'default' => false,
	// see T203179
	'fixcopyrightwiki' => true,
],

Could this be the cause? (ping @Jdforrester-WMF)

H78c67c added a comment.EditedFeb 25 2020, 10:50 AM

There is another issue which I am not sure if it is related, but the Wiki Loves Folklore CentralNotice is in Urdu when the interface language is English.

It appears to be a problem with https://meta.wikimedia.org/w/index.php?title=CNBanner:Wikiloveslove2020-text/en-gb&oldid=19754638, not related to this issue.

https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/552549/2/wmf-config/InitialiseSettings.php deployed recently for T238803 removed:

'wgULSLanguageDetection' => [
	'default' => false,
	// see T203179
	'fixcopyrightwiki' => true,
],

Could this be the cause? (ping @Jdforrester-WMF)

ULSLanguageDetection defaults to true in its extension.json, so that seems likely.

Change 574743 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Reinstate wgULSLanguageDetection setting

https://gerrit.wikimedia.org/r/574743

Is this causing cache pollution? If so, this should be merged ASAP to limit the damage of that.

Mardetanha added a subscriber: Mardetanha.

Either caches are being polluted, or they are getting split (Vary: Accept-Language) or they are being bypassed.

Since sites are still up, likely not the last one. Since I am getting Finnish, probably not the first one either, so maybe cache splitting is happening?

Change 574743 merged by jenkins-bot:
[operations/mediawiki-config@master] Reinstate wgULSLanguageDetection setting

https://gerrit.wikimedia.org/r/574743

I just did some quick testing with curl. At a glance, it looks like varnish is respecting the vary: accept-language header and splitting cache.

Mentioned in SAL (#wikimedia-operations) [2020-02-25T14:15:52Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:574743|Reinstate wgULSLanguageDetection setting (T246071)]] (duration: 01m 03s)

Hm. The config change was deployed, and it seemed to do the right thing on mwdebug1001, but I’m still seeing English on https://zh-yue.wikipedia.org/wiki/%E9%A0%AD%E7%89%88.

Hm. The config change was deployed, and it seemed to do the right thing on mwdebug1001, but I’m still seeing English on https://zh-yue.wikipedia.org/wiki/%E9%A0%AD%E7%89%88.

Works for me when logged out.

Yup, now it’s Chinese too. ruwiki (from duplicate task T246095) shows English UI for me at the moment, let’s see if that also fixes itself I guess?

Current status seems to be that the appservers behave correctly now (it may have taken them a bit to react to the change, it’s not clear); for a while, there seemed to be cached responses for some Accept-Language values in Varnish, which weren’t getting removed by their own, but now those appear to be gone as well, possibly having timed out. I can’t reproduce the issue on any page anymore.

It looks like we hit T236104: Cache of wmf-config/InitialiseSettings often 1 step behind. Looking at graphs things seems to be returning to normal, at least avg. response time and application server cluster load.

Started 00:53Z
Fix really applied: 14:35Z

Eurgh. Sorry about this, all. I assumed ULS's defaults were deploy-safe and didn't check.

Incident documentation is here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20200225-mediawiki_interface_language

As the issue seems to be fixed, shall we close this task? Or should it stay open for the User-notice or something?

Incident documentation is here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20200225-mediawiki_interface_language

As the issue seems to be fixed, shall we close this task? Or should it stay open for the User-notice or something?

None of the affected users will ever read Tech/News, or have any way to converse with someone who does. I don't think it'd add value.

Jdforrester-WMF closed this task as Resolved.Feb 25 2020, 4:03 PM

None of the affected users will ever read Tech/News, or have any way to converse with someone who does. I don't think it'd add value.

(I don’t necessarily disagree, I just saw that @Nikerabbit had added the tag.)

Incident documentation is here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20200225-mediawiki_interface_language

As the issue seems to be fixed, shall we close this task? Or should it stay open for the User-notice or something?

None of the affected users will ever read Tech/News, or have any way to converse with someone who does. I don't think it'd add value.

Some of them might (not everyone is logged in all the time), and there might be discussions on wikis about this behavior. I'd think it would be useful to let them know that this was an accident and not a planned changed, but I'll let the tech news writers decide.

Johan added a subscriber: Johan.Feb 27 2020, 10:51 AM

(Either way – I have no strong opinion on including it or not – I don't think tickets need to stay open for Tech News.)

Jony added a subscriber: Jony.Mar 4 2020, 5:37 AM
Nirmos added a subscriber: Nirmos.Mar 4 2020, 6:53 PM

The post-mortem at https://wikitech.wikimedia.org/wiki/Incident_documentation/20200225-mediawiki_interface_language states

Reported by users at phabricator:T246071 and other tasks. There were no alerts as far as Lucas Werkmeister (WMDE) is aware.

and

Impact was not clear from the beginning, since the additional load was not significant to trigger any alerts (even through average app server response time jumped 33%)

and suggests

If human only, an actionable should probably be "add alerting".

Should a task for that be created?

Well, I have no idea what the alerting should be… adding an alert specifically for “Accept-Language is used” doesn’t sound reasonable to me.

If the load had increased enough to be a problem in itself, I assume there would have been alerts.