Page MenuHomePhabricator

Varnish: "Cacheable object with Set-Cookie found" on some rare action=raw responses
Open, Needs TriagePublic

Description

Despite the defense layers in MediaWiki (via OutputPage::sendCacheControl, and HeaderCallback) we have a a dozen responses where Varnish is protecting us from cache poisoning by disabling caching for an object that claims to be cachable due to it setting cookies.

We wrote this defense layer for Varnish in 2020 (T264378, T256395) after an incident in which session cookies where accidentally cached and served out to someone else, causing impersonation issues.

We have about 256M (2M x 1/128 sampled) requests per day to /w/index.php?action=raw. https://w.wiki/Srzi

Screenshot 2026-07-29 at 17.12.51.png (1,694×704 px, 85 KB)

We have exactly 11 requests over the same period, that emit this warning. https://logstash.wikimedia.org/goto/f7220207d8cf9b0e209d78571fffbff2

Screenshot 2026-07-29 at 17.31.39.png (2,060×920 px, 145 KB)

Cacheable object with Set-Cookie found!
  beresp.status: 200
  beresp.was_304: true
  bereq.http.Host: meta.wikimedia.org
  bereq.url: /w/index.php?action=raw&ctype=text/javascript&title=MediaWiki:Wikiminiatlas.js
  Cache-Control: public, s-maxage=300, max-age=1209600
  Set-Cookie: ***
  ReqId: 00e52db0-2823-4082-a2c5-2ce167311649
  X-Cache-Int: cp6016 miss

I've omitted the cookie for security reasons, but in this case it's actually fine:

Set-Cookie: metawikiUserID=deleted; expires=Thu, 01 Jan 1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=None

They're all like this. It's not so much the creation of a cookie but the deletion of a cookie.

I have a hunch this may have to do with cross-domain requests not appearing quite valid and so MediaWiki wants to delete it. And then I'm guessing our set-cookie detector is ignoring cases where the cookie is deleted? They look like normal set-cookie headers just with a date in the past and our dummy value of "deleted" so it's not obvious why we'd ignore it.

PS: Note that this task is not about violations in MediaWiki extensions where the aforementioned defense layers in MediaWiki core succesfully catch and correct the cache header before Varnish sees it. For those, see T285210 and T149873.

Event Timeline

Looking at responses for anything else, there are only 2 over the same period.

Logstash: "Cacheable object with Set-Cookie" AND NOT "action=raw"

Cacheable object with Set-Cookie found!
 beresp.status: 200 beresp.was_304: true
 bereq.http.Host: commons.wikimedia.org bereq.url: /wiki/Portada?uselang=es
 Cache-Control: s-maxage=3600, must-revalidate, max-age=0 Set-Cookie: commonswikiUserID=deleted; expires=Thu, 01 Jan 1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=None
 Server: ATS/9.2.13 ReqId: 6a553348-e9e4-4388-89d9-3c36abb69af0 X-Cache-Int: cp7002 miss
Cacheable object with Set-Cookie found!
 beresp.status: 200 beresp.was_304: true
 bereq.http.Host: pl.wikipedia.org bereq.url: /wiki/James_Trafford
 Cache-Control: s-maxage=745295, must-revalidate, max-age=0 Set-Cookie: plwikiUserID=deleted; expires=Thu, 01 Jan 1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=None
 Server: ATS/9.2.13 ReqId: c86170a9-aef6-4999-af7c-a5dd4af6e4ae X-Cache-Int: cp3066 miss

Those two look like the same problem, so it's probably not caused by action=raw but rather by a race condition that happens to be more common there than elsewhere. This supports my theory that it is caused by cross-domain requests shortly after a central login.

The scenario is like this: You perform a central login on wiki A. This will in the background perform auto-login for wikis B and C. While that is happening, you open a second tab to wiki C. In theory anyone could do that, but it's unlikely as the window for it is only a few seconds, and humans generally can't do two things at once.

However, if we don't think about it as opening a second tab, but as making a cross-domain request, then it's quite a normal scenario. If you have a gadget or user script enabled on wiki A that imports code via action=raw from wiki C, then you get the following reliably every time you login:

  • Log in via wiki A
  • On the first pageview after log in:
    • CentralAuth JavaScript code will perform an auto-login for wikis B and C
    • Your own JavaScript code imports code via action=raw also from wiki C. These can easily overlap or race with each other.

We came across this before last year at T385310#11436185 with the same race condition.

I don't yet know what factor is causing this to happen only 13 times a day (11 action=raw, 2 regular navivgations). There's some other funnel going on making this rare.

These are both session cookie deletions, so not central login.

Probably a session expiry (or logout on a different device) - the client sends session cookies which are invalid, SessionManager handles that as an anonymous request, but also clears the session cookies.