Page MenuHomePhabricator
Create Task
Maniphest T209021

ATS backend-side request-mangling
Closed, ResolvedPublic
Actions

Description

We need to ensure that ATS provides the same functionality currently provided by the varnish-be layer. This includes:

  • not caching files with explicit Content-Length >= ~1GB
  • Disable streaming if no Content-Length present
  • normalize PURGE requests paths T210295
  • set X-MediaWiki-Original for thumbnails based on the original request URL
  • mobile/restbase hostname/url mangling
  • ensure Set-Cookie responses are not cached by ATS and not cacheable by other layers

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
ATS: do not cache files bigger than 1GBEmaoperations/puppetproduction+26 -0
ATS: set cache_responses_to_cookies to 1Emaoperations/puppetproduction+14 -10
ATS: do not cache cookie responsesEmaoperations/puppetproduction+41 -0
ATS: MediaWiki and RestBASE request manglingEmaoperations/puppetproduction+76 -0
ATS: set X-MediaWiki-Original for upload.w.oEmaoperations/puppetproduction+64 -0
ATS: allow specifying remap-rule-specific Lua scriptsEmaoperations/puppetproduction+9 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

• ema created this task.Nov 8 2018, 8:15 AM
Restricted Application added a project: SRE. · View Herald TranscriptNov 8 2018, 8:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ema triaged this task as Medium priority.Nov 8 2018, 8:16 AM
• ema mentioned this in T207048: ATS production-ready as a backend cache layer.
• ema added a parent task: T207048: ATS production-ready as a backend cache layer.
• ema updated the task description. (Show Details)
• ema moved this task from Backlog to Caching on the Traffic board.Nov 8 2018, 8:46 AM
gerritbot subscribed.Nov 12 2018, 12:45 PM

Change 472971 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: allow specifying remap-rule-specific Lua scripts

https://gerrit.wikimedia.org/r/472971

gerritbot added a project: Patch-For-Review.Nov 12 2018, 12:45 PM

Change 472972 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: set X-MediaWiki-Original for upload.w.o

https://gerrit.wikimedia.org/r/472972

gerritbot added a comment.Nov 12 2018, 1:33 PM

Change 472971 merged by Ema:
[operations/puppet@production] ATS: allow specifying remap-rule-specific Lua scripts

https://gerrit.wikimedia.org/r/472971

gerritbot added a comment.Nov 12 2018, 1:33 PM

Change 472972 merged by Ema:
[operations/puppet@production] ATS: set X-MediaWiki-Original for upload.w.o

https://gerrit.wikimedia.org/r/472972

• ema added a comment.Nov 12 2018, 1:46 PM

X-MediaWiki-Original is now being set by ATS with a remap rule specific to upload.wikimedia.org:

$ curl -sIH "Host: upload.wikimedia.org" http://cp1072.eqiad.wmnet:3129/wikipedia/commons/thumb/3/37/Home_Albert_Einstein_1895.jpg/200px-Home_Albert_Einstein_1895.jpg | grep Original
X-MediaWiki-Original: /wikipedia/commons/3/37/Home_Albert_Einstein_1895.jpg
• ema updated the task description. (Show Details)Nov 23 2018, 4:18 PM
• ema closed subtask T210295: ATS path normalization as Resolved.Nov 26 2018, 4:04 PM
gerritbot added a comment.Dec 3 2018, 9:28 AM

Change 477221 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: MediaWiki and RestBASE request mangling

https://gerrit.wikimedia.org/r/477221

• ema updated the task description. (Show Details)Dec 3 2018, 9:32 AM
gerritbot added a comment.Dec 3 2018, 9:32 AM

Change 477221 merged by Ema:
[operations/puppet@production] ATS: MediaWiki and RestBASE request mangling

https://gerrit.wikimedia.org/r/477221

gerritbot added a comment.Dec 3 2018, 10:55 AM

Change 477240 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: do not cache cookie responses

https://gerrit.wikimedia.org/r/477240

gerritbot added a comment.Dec 3 2018, 11:10 AM

Change 477240 merged by Ema:
[operations/puppet@production] ATS: do not cache cookie responses

https://gerrit.wikimedia.org/r/477240

gerritbot added a comment.Dec 3 2018, 1:13 PM

Change 477262 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: set cache.cache_responses_to_cookies to 1

https://gerrit.wikimedia.org/r/477262

gerritbot added a comment.Dec 3 2018, 1:34 PM

Change 477262 merged by Ema:
[operations/puppet@production] ATS: set cache_responses_to_cookies to 1

https://gerrit.wikimedia.org/r/477262

• ema updated the task description. (Show Details)Dec 3 2018, 2:24 PM
gerritbot added a comment.Dec 6 2018, 11:42 AM

Change 477991 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: do not cache files bigger than 1GB

https://gerrit.wikimedia.org/r/477991

gerritbot added a comment.Dec 6 2018, 12:16 PM

Change 477991 merged by Ema:
[operations/puppet@production] ATS: do not cache files bigger than 1GB

https://gerrit.wikimedia.org/r/477991

• ema updated the task description. (Show Details)Dec 6 2018, 12:21 PM
• ema closed this task as Resolved.Dec 6 2018, 12:40 PM

All the functionalities currently provided by our varnish backends in terms of request/response mangling have been implemented, with two exceptions:

  1. ATS should (hopefully!) not be affected by the same scalability issues that plague the varnish file storage backend (T145661). We can thus cache files with CL < 1024
  2. There does not seem to be a straightforward way to "disable streaming" in ATS, ie stall clients while fetching the whole response from the origin server, store it in cache, and then respond to all waiting clients. This is what we currently do in varnish-backend-land to ensure that all responses have a Content-Length. ATS does however add missing CLs to all cache hits, including those happening while the object is being written to cache (a concept called Read-While-Writer). It should thus be fine to let ATS serve cache misses without Content-Length, and change the varnish frontend logic from "do not cache objects with CL>N" to "do not cache objects with CL>N or without CL".
ssingh mentioned this in T411043: Revisit the 1GB cache size limit for ATS.Nov 25 2025, 6:59 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 25 2025, 7:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits