/static generates (and caches!) redirect loops on cache-miss
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Jul 2 2015, 2:58 AM

Description

These two urls were both accessed shortly before the files in question were synchronised across all servers.

https://www.wikimedia.org/static/images/wmf-2x.png
https://www.wikimedia.org/static/images/wmf.png?foo

Both are now stuck in cache as a redirect loop.

$ curl -i 'https://www.wikimedia.org/static/images/wmf-2x.png'
HTTP/1.1 301 Moved Permanently
Server: nginx/1.9.2
Date: Thu, 02 Jul 2015 02:56:30 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 258
Connection: keep-alive
X-Powered-By: HHVM/3.3.0-static
Location: https://www.wikimedia.org/static/images/wmf-2x.png
Cache-Control: max-age=2592000
Expires: Sat, 01 Aug 2015 02:52:28 GMT
Vary: X-Forwarded-Proto
X-Varnish: 3588321948, 1745654129, 2307457473 2307396811
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Age: 242
X-Cache: cp1052 miss (0), cp3013 miss (0), cp3004 frontend hit (147)
Strict-Transport-Security: max-age=15768000
X-Analytics: https=1X-Analytics: https=1


$ curl -i 'https://www.wikimedia.org/static/images/wmf.png?foo'
HTTP/1.1 301 Moved Permanently
Server: nginx/1.9.2
Date: Thu, 02 Jul 2015 02:56:56 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 259
Connection: keep-alive
X-Powered-By: HHVM/3.3.0-static
Location: https://www.wikimedia.org/static/images/wmf.png?foo
Cache-Control: max-age=2592000
Expires: Sat, 01 Aug 2015 02:53:05 GMT
Vary: X-Forwarded-Proto
X-Varnish: 1384589162, 657231649, 2307464143 2307406105
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Age: 231
X-Cache: cp1055 miss (0), cp3006 miss (0), cp3004 frontend hit (128)
Strict-Transport-Security: max-age=15768000
Set-Cookie: GeoIP=*
X-Analytics: https=1

Other urls such as these two are both fine:

https://www.wikimedia.org/static/images/wmf-2x.png?bar
https://www.wikimedia.org/static/images/wmf.png

Details

	Subject	Repo	Branch	Lines +/-
	Replace static-hash with hostname normalization	operations/puppet	production	+6 -29
	Fix /static hashing by forcing www.wm.o hostname	operations/puppet	production	+8 -3

Customize query in gerrit

Related Objects

Mentioned In: rOPUP2b06db20e874: Replace static-hash with hostname normalization
rOPUP7ae5a4b9fb46: Fix /static hashing by forcing www.wm.o hostname

Event Timeline

Krinkle created this task.Jul 2 2015, 2:58 AM

Krinkle raised the priority of this task from to Needs Triage.

Krinkle updated the task description. (Show Details)

Krinkle added projects: Varnish, acl*sre-team.

Krinkle subscribed.

Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptJul 2 2015, 2:58 AM

Krinkle set Security to None.Jul 2 2015, 2:58 AM

Krinkle added subscribers: ori, BBlack.

I've send a squid purge to both of them to resolve the immediate issue.

Krinkle updated the task description. (Show Details)Jul 2 2015, 3:02 AM

So, after debugging this on IRC with Krinkle, the net of the problem is this:

Varnish in static-hash.inc.vcl is completely hostname-agnostic about /static paths by design, because that's efficient since it should be the same on all hostnames. In practice, what this means is that the initial cache miss for a given /static/foo is requested from the applayer with the hostname supplied by the user, and the result is re-used as a cache hit for all requests for /static/foo regardless of request hostname.
If an apache-level redirect is defined for e.g. cz.wikipedia.org -> cs.wikipedia.org, the applayer returns a 301 with the full path copied for all requests to the redirected domain.
Therefore if the first hit on a static URL is for a redirect domain, it's cached as that redirect for all domains, creating redirect loops. Example, using a non-existent actual file:

bblack-mba:puppet bblack$ curl -sv 'https://cz.wikipedia.org/static/images/wmf-2x.pngxxx' 2>&1 >/dev/null |egrep 'Moved|Location'
< HTTP/1.1 301 Moved Permanently
< Location: https://cs.wikipedia.org/static/images/wmf-2x.pngxxx
bblack-mba:puppet bblack$ curl -sv 'https://cs.wikipedia.org/static/images/wmf-2x.pngxxx' 2>&1 >/dev/null |egrep 'Moved|Location'
< HTTP/1.1 301 Moved Permanently
< Location: https://cs.wikipedia.org/static/images/wmf-2x.pngxxx

For now, my proposed fix is that we simply force the virtual request hostname to en.wikipedia.org for all requests to /static.

Change 222242 had a related patch set uploaded (by BBlack):
Fix /static hashing by forcing enwiki hostname

https://gerrit.wikimedia.org/r/222242

gerritbot added a project: Patch-For-Review.Jul 2 2015, 3:45 AM

This problem surfaced due to incoming requests for /static using https://wikimedia.org, which is a redirect to https://www.wikimedia.org on cache-miss.

But on cache-hit, /static is served as-is before any domain redirects.

Change 222242 merged by BBlack:
Fix /static hashing by forcing www.wm.o hostname

https://gerrit.wikimedia.org/r/222242

BBlack mentioned this in rOPUP7ae5a4b9fb46: Fix /static hashing by forcing www.wm.o hostname.Jul 2 2015, 4:16 AM

Change 222249 had a related patch set uploaded (by BBlack):
Replace static-hash with hostname normalization

https://gerrit.wikimedia.org/r/222249

Change 222249 merged by BBlack:
Replace static-hash with hostname normalization

https://gerrit.wikimedia.org/r/222249

ori mentioned this in rOPUP2b06db20e874: Replace static-hash with hostname normalization.Jul 2 2015, 4:58 AM

Krinkle closed this task as Resolved.Jul 12 2015, 11:49 AM

Krinkle claimed this task.

Krinkle triaged this task as High priority.

Aklapper added a project: Traffic.May 4 2016, 9:14 AM

/static generates (and caches!) redirect loops on cache-missClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

/static generates (and caches!) redirect loops on cache-miss
Closed, ResolvedPublic
Actions