@Steinsplitter: If this task is different from T58464, should this task depend on T58464 with status "stalled"?

In T161517#3134091, @Aklapper wrote:

@Steinsplitter: If this task is different from T58464, should this task depend on T58464 with status "stalled"?

I don't think it is "stalled". As far i can see the config var ($wgULSAnonCanChangeLanguage) exists yet, but needs to be set so anons can change the language with ULS instead of using the "Change language" in the navbar and we can remove the old js hack which does the same in a worser way.

In T161517#3136977, @Jdforrester-WMF wrote:

It's stalled because the way ULS works for anons is incompatible with Wikimedia's hosting infrastructure.

Wikimedia's hosting infrastructure doesn't yet support this requirement of the users of Wikimedia sites.

Varying the interface with the current infrastructure would take down the sites.

It probably wouldn't, per T58464#3131083.

Wikimedia avoids having to spend many millions of dollars on servers each year by serving the exact-same content to all anonymous users.

Yes, there is definitely an additional cost for this feature, but unlikely in the scale of millions.

Scaling this up to (around) 300 copies of every page means much worse performance for all, and hundreds of extra servers. It's very stalled. :-(

The assumption that we must make 300 copies of every page is not true because 1) not all users will set a language 2) there is a trade-off between caching and serving uncached hits that can be tweaked. 3) not all pages are going to be viewed in all supported languages.

As far as I understand the JS hack circumvents the caching infrastructure for all readers with the ?uselang=xx parameter. Isn't this the worst case?
So every technique to cache would improve the situation.

I do not have any statistics/numbers so the following is based on my some assumptions to keep the cache size in a sane state:

• Limit the cache to the 10 most important languages.

• Limit the cache duration to 5 days (instead of 30 currently?). Most readers are looking for an image, browsing the category(tree) and when found an image they leave the site. These file pages would be cached then for 5 days for the case the same user come back or another user (in the same language) is looking for similar files.

If I am on the totally wrong path kill my comment with fire :-)

This report is not waiting for input, so it's not stalled according to the definition at https://www.mediawiki.org/wiki/Bug_management/Bug_report_life_cycle

Change 345342 had a related patch set uploaded (by Steinsplitter):
[operations/mediawiki-config@master] Set wmgULSAnonCanChangeLanguage true for commonswiki.

https://gerrit.wikimedia.org/r/345342

Adding Traffic and myself and @ema to this. I don't think we've been aware of the uselang hack or its mechanics before (why did ?uselang=foo trigger uncacheability in the first place? query params vary the cache by default, it would've been fine and preferable to leave it cacheable...).

Similar comments apply about the ULS cookie, described at the top as So, for Commons, enabling anon USL (and making pages with the ULS cookie set not cached) would should be fine. Our caches would not, by default, honor the ULS cookie as something that makes pages uncacheable (and thus, the shared caching of these pages would become polluted by random users' language choices, and/or their language choices would fail to be honored by existing cache hits).

At the very least, implementing that plan as stated requires additional support from Traffic to make custom hacks to our cookie-handling code in Varnish (e.g. we could treat site-level ULS cookies the same as we do login-session cookies to make those pageviews uncacheable). But again, as with ?uselang, I don't think it would be desirable to make these uncacheable in the first place. What we'd really want is to Vary the cache on the ULS cookie and leave the pages cacheable. This is considerably more difficult than what we could've done for cacheability with ?uselang, and would also require some Varnish-level work (our support for true per-cookie-value variance is basically non-existent at this point).

Adding Traffic and myself and @ema to this. I don't think we've been aware of the uselang hack or its mechanics before (why did ?uselang=foo trigger uncacheability in the first place? query params vary the cache by default, it would've been fine and preferable to leave it cacheable...).

The main reason I imagine is because this leaves the pages cached but unpurgable. Given that only the canonical urls from Title::getCdnUrls() are purged after an edit.

Historically, we have indeed allowed arbitrary query parameter combinations and simply let them be cached with the drawback of not getting purged automatically.

However it does seem, contrary what I expected, uselang does override this:

https://commons.wikimedia.org/wiki/Main_Page?uselang=en-gb
cache-control:private, s-maxage=0, max-age=0, must-revalidate
backend-timing:D=113733 t=1490803262893132
date:Wed, 29 Mar 2017 16:01:03 GMT
expires:Thu, 01 Jan 1970 00:00:00 GMT
last-modified:Wed, 29 Mar 2017 14:16:25 GMT
x-cache:cp1065 pass, cp2023 pass, cp4008 pass, cp4008 pass
x-cache-status:pass

If we were to remove this override (should be safe as far as I can tell), the only drawback would be that users being pulled into this feature on Commons may see 1-7 day old stale versions of pages.

I don't think we've been aware of the uselang hack or its mechanics before

The documentation for developers was "originally" provided at https://www.mediawiki.org/wiki/User:Pfctdayelise/Manual:Uselang_hack (since around the time this method was cemented i.e. after the first time it was accidentally broken).

I think that's a different uselang hack.

In T161517#3144389, @matmarex wrote:

I think that's a different uselang hack.

Yepp, This one (AnonymousI18N) is used on commons since 2010/11.

@ema will this be fixed soon? If not i have to fix stuff & update the MediaWiki message on commons (so that users won't be confused any longer). Thanks :)

In T161517#3153014, @Steinsplitter wrote:

@ema will this be fixed soon? If not i have to fix stuff & update the MediaWiki message on commons (so that users won't be confused any longer). Thanks :)

IMHO the simplest solution here would be to do as @Krinkle mentioned: make ?uselang=foo cacheable, with the drawback that language-specific versions of the pages won't get purged by default and users might end up accessing stale versions after an edit. By using cache-control, however, you can tune "how stale" you want the objects to be.

If instead we have to Vary the cache on the ULS cookie and leave the pages cacheable as @BBlack said, that's gonna take some time.

This is the only pending question, isn't it?

In T161517#3140732, @Krinkle wrote:

If we were to remove this override (should be safe as far as I can tell), the only drawback would be that users being pulled into this feature on Commons may see 1-7 day old stale versions of pages.

I'm not completely sure, but serving 1-7 days old pages to unregistered users is probably an acceptable cost to pay.

My comment T161517#3142987 was meant to address this question indirectly: we know that uselang is used for various kinds of content alteration, so serving stale content has larger implications than it normally would. But we can consider re-introducing the override later if needed, no?

This task has gotten a bit confusing. Stepping back a bit from the specific case of Commons (because I think the same issues apply everywhere?)... let me try to recap here a little, and correct me please if I've gotten some of this wrong:

1. Language-switching for logged-in users - Works, using ULS cookies, because logged-in traffic is completely uncacheable/uncached anyways (unfortunately, but that's a whole different ball of wax)
2. Language-switching for anons via uselang javascript hack - Dated and undesirable, but "works". Cache-level issues are (a) it's currently emitted as uncacheable, which is not great for hitrate/perf and (b) If it were allowed to be cacheable it would function fine, but wouldn't get invalidated alongside the main URL(s) when the article is purged for edit (etc). Also note it's not 1-7 days of potential staleness here, it's more like 1-4 days with a strong skew towards the low end. We have an outer corner-case of ~4 days on overall cache infrastructure TTL (assuming worst-case user location and a very hot article), with common real-world cases caching for closer to ~24h (or less).
3. Language-switching for anons via ULS cookies - Theoretically works on the MediaWiki end of things, but turning this on will cause big problems in production because our caches do not vary their object outputs on this cookie correctly, thus there will be cross-pollution of language outputs for users. Fixing this is something that needs separate tasking and some work on the Traffic end of things (or maybe even some design discussion about whether this is even the best way to select output languages in the first place? Should this whole discussion be considered in light canonicals and their outputs, the Accept-Language header (which yes, has some usability issues in UAs I think), search engines, etc? But regardless, this isn't trivial).

I think if there's a desire to get anon language-switching deployed in a shorter timeframe than it might takes to sort out (3), then (2) is probably a sane temporary recourse. It'd be nice to make these ?uselang outputs cacheable as well, perhaps limiting the total MediaWiki-specified max-age to a lower-than-default value of 1d to ensure they're never more-stale than that.

In T161517#3380185, @BBlack wrote:

This task has gotten a bit confusing.

I wrote a proposal a while ago in T58464: Allow anonymous users to change interface language on Wikimedia wikis with ULS that summarises some parts.

I think if there's a desire to get anon language-switching deployed in a shorter timeframe than it might takes to sort out (3)

Can you clarify what needs to be sorted out to move forward with "manual language selection for anonymous users using ULS". How much effort is it? Per my understanding, it is sufficient to make the caches either vary based on the language cookie value (with the issue you note in your comments) or make requests containing the cookie uncacheable.

I understand that there are complexities with the Accept-Languages header and that it is possible to spend time re-evaluating all the implementation choices that have and haven't been made. However, for all these year we have been missing the basic feature of manual language selection that is available for all users. I don't want to delay it further. I believe my proposal for the first step is doable, worth it even if we don't go further, and does not block future changes to improve the caching related issues. Please let me know if you disagree.

Ok I think I was confused as to the state of the uselang hack. It looks like it already works, in uncacheable form, on all wikis? When I try it on enwiki with uselang set to es, I get an uncacheable copy of the article with the UI header stuff in Spanish, the content-language response header still set to en, and of course the article text still in en. On that front, it would still be an improvement if uselang outputs were cacheable, IMHO.

Can you clarify what needs to be sorted out to move forward with "manual language selection for anonymous users using ULS". How much effort is it? Per my understanding, it is sufficient to make the caches either vary based on the language cookie value (with the issue you note in your comments) or make requests containing the cookie uncacheable.

Yes, this. If we do neither of these things at the Varnish layer, everything will get language-broken on deploy. The Vary-based solution is the only acceptable one at this time (we don't want to make even more anon fetches uncacheable, especially as the ULS selection probably doesn't actually change the output of all the URLs we'd be disabling caching on, and is likely to be widely-used on the popular cross-language wikis like commons and wikidata), but it's a thorny mess.

The key underlying problem is that we have debt around Cookie-varying. Right now Vary: Cookie is (at this point, effectively) used as a signal from the application layer to the caches that content should be uncacheable for logged-in users. When we see that response header in combination with a user session/token cookie on the input side, we disable caching. This is basically how logged-in users work at all, and it's pretty suboptimal and doesn't allow much flexibility. If we want to independently Vary on more than one Cookie, we'd need signals about both conditions independently. @tstarling had started solving this with his X-Vary-Options proposal under Squid many years ago, but that was lost in the transition to Varnish slightly fewer years ago, when we fell back to the simpler "Vary: Cookie means logged-in-user-specific output which is uncacheable" hack. As ugly as our current solution is, it "works" for the present and there aren't other pressing issues around it that seem to beat the priorities and/or timelines of efforts to move away from Varnish entirely.

We could re-implement the XVO header support in Varnish, although at a glance I think it needs some expansion. It seems (again, at a quick look) to support either varying fully on a header, or varying on the boolean presence of specific substrings, but nothing that quite encapsulates an idea like "Split the Cookie header into k=v pairs, and then fully-vary on the values of the key foo"? I also recall @faidon at one point mentioning either an RFC or W3C proposed standard that seemed to overlap with our XVO that we might want to transition to, but I can't for the life of me dredge up a reference about it anywhere. Vary itself is deeply integrated in Varnish, so (parsing issues aside) I doubt XVO can be easily implemented at the VCL level without incredible hacks around creating alternate fake request headers matching the actual varying conditions, and a faked Vary header that references them.

It would probably be best-tackled as a vmod project, but before investing in it heavily we'd really want to know whether XVO is our best option (or if that other newer and more-standard option exists and works?), and on another level whether ULS using cookies to indicate the user's UI language is really our best architectural path forward before we invest so significantly in supporting it. We could also skip over the full depth of a solution like XVO and instead implement something more like "X-Vary-Cookie-Keys: language,session,token", but even that route isn't simple to implement, as our current session/token cookie keys need regexes, which don't encode in headers easily (or at least, end-anchored substrings, that might be simpler to support).

Another path that occurs to me is that we could leave the session/token hacks as is (explicit support via regexes directly in VCL code), and create a new "X-Vary-Cookie-Keys: language" which is implicitly understood to not cover the session case. In this case we'd have distinct signals from the applayer in the form of "Vary: Cookie" for sessions and "X-Vary-Cookie-Keys: language" for ULS. This isn't ideal, but it's a lot less work than other options, and doesn't require any changes to anything related to centralauth or sessions, just ULS-language-differentiating outputs. It can probably be reasonably implemented in VCL, but still not completely trivially. We'd basically ignore this when it overlaps with logged-in-user uncacheability, and then temporarily strip the cookie value down to just the language k=v and inject a false Vary:Cookie in the anon+ULS case. It's still more VCL tech debt to support in future solutions. It would require some work on the ULS side as well: any URL output which pays attention to the language cookie from ULS (which I assume involves some code hook already), must set the "X-Vary-Cookie-Keys: language" header in its output (always, even if there was no input Cookie present). URLs that don't vary their output on the language cookie should not set the header.

All of these things probably constitute one of the reasons why we've been missing this basic feature for all these years. Note also the fascinating thread from 1997 on this same topic, and the sorts of solutions and their ills that are discussed (pretty much nothing has changed on this front since, except that Cookie has become almost universally polluted with non-origin keys in practice): http://lists.w3.org/Archives/Public/ietf-http-wg-old/1997MayAug/0334.html .

Thanks for the detailed reply which explains why it isn't so simple as I thought. I need to read it again a few times to understand it better. .

I can commit to doing changes in ULS if those are needed, but the number of alternatives and their consequences makes me wonder whether this ought to go through our RfC process for wider input. Thoughts?

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

This also applies to wikidata.

Change 345342 abandoned by Nikerabbit:

[operations/mediawiki-config@master] Set wmgULSAnonCanChangeLanguage true for commonswiki.

Reason:

No point keeping this patch open until the blockers are resolved.

https://gerrit.wikimedia.org/r/345342

Wikidata does not use uselang hack: this task is specific to Commons. See T58464 for a global task (all wikis, including Wikidata); or T352645 for Wikidata eventual specificities.