Page MenuHomePhabricator

ATS backend-side request-mangling
Closed, ResolvedPublic

Description

We need to ensure that ATS provides the same functionality currently provided by the varnish-be layer. This includes:

  • not caching files with explicit Content-Length >= ~1GB
  • Disable streaming if no Content-Length present
  • normalize PURGE requests paths T210295
  • set X-MediaWiki-Original for thumbnails based on the original request URL
  • mobile/restbase hostname/url mangling
  • ensure Set-Cookie responses are not cached by ATS and not cacheable by other layers

Event Timeline

ema created this task.Nov 8 2018, 8:15 AM
Restricted Application added a project: Operations. · View Herald TranscriptNov 8 2018, 8:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema moved this task from Triage to Caching on the Traffic board.Nov 8 2018, 8:46 AM

Change 472971 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: allow specifying remap-rule-specific Lua scripts

https://gerrit.wikimedia.org/r/472971

Change 472972 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: set X-MediaWiki-Original for upload.w.o

https://gerrit.wikimedia.org/r/472972

Change 472971 merged by Ema:
[operations/puppet@production] ATS: allow specifying remap-rule-specific Lua scripts

https://gerrit.wikimedia.org/r/472971

Change 472972 merged by Ema:
[operations/puppet@production] ATS: set X-MediaWiki-Original for upload.w.o

https://gerrit.wikimedia.org/r/472972

X-MediaWiki-Original is now being set by ATS with a remap rule specific to upload.wikimedia.org:

$ curl -sIH "Host: upload.wikimedia.org" http://cp1072.eqiad.wmnet:3129/wikipedia/commons/thumb/3/37/Home_Albert_Einstein_1895.jpg/200px-Home_Albert_Einstein_1895.jpg | grep Original
X-MediaWiki-Original: /wikipedia/commons/3/37/Home_Albert_Einstein_1895.jpg
ema updated the task description. (Show Details)Nov 23 2018, 4:18 PM

Change 477221 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: MediaWiki and RestBASE request mangling

https://gerrit.wikimedia.org/r/477221

ema updated the task description. (Show Details)Dec 3 2018, 9:32 AM

Change 477221 merged by Ema:
[operations/puppet@production] ATS: MediaWiki and RestBASE request mangling

https://gerrit.wikimedia.org/r/477221

Change 477240 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: do not cache cookie responses

https://gerrit.wikimedia.org/r/477240

Change 477240 merged by Ema:
[operations/puppet@production] ATS: do not cache cookie responses

https://gerrit.wikimedia.org/r/477240

Change 477262 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: set cache.cache_responses_to_cookies to 1

https://gerrit.wikimedia.org/r/477262

Change 477262 merged by Ema:
[operations/puppet@production] ATS: set cache_responses_to_cookies to 1

https://gerrit.wikimedia.org/r/477262

ema updated the task description. (Show Details)Dec 3 2018, 2:24 PM

Change 477991 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: do not cache files bigger than 1GB

https://gerrit.wikimedia.org/r/477991

Change 477991 merged by Ema:
[operations/puppet@production] ATS: do not cache files bigger than 1GB

https://gerrit.wikimedia.org/r/477991

ema updated the task description. (Show Details)Dec 6 2018, 12:21 PM
ema closed this task as Resolved.Dec 6 2018, 12:40 PM

All the functionalities currently provided by our varnish backends in terms of request/response mangling have been implemented, with two exceptions:

  1. ATS should (hopefully!) not be affected by the same scalability issues that plague the varnish file storage backend (T145661). We can thus cache files with CL < 1024
  2. There does not seem to be a straightforward way to "disable streaming" in ATS, ie stall clients while fetching the whole response from the origin server, store it in cache, and then respond to all waiting clients. This is what we currently do in varnish-backend-land to ensure that all responses have a Content-Length. ATS does however add missing CLs to all cache hits, including those happening while the object is being written to cache (a concept called Read-While-Writer). It should thus be fine to let ATS serve cache misses without Content-Length, and change the varnish frontend logic from "do not cache objects with CL>N" to "do not cache objects with CL>N or without CL".