| Krinkle | |
| Mar 5 2020, 7:29 PM |
| F31672603: Screenshot from 2020-03-09 11-19-05.png | |
| Mar 9 2020, 10:20 AM |
| F31667331: Screenshot 2020-03-05 at 19.23.18.png | |
| Mar 5 2020, 7:29 PM |
| F31667327: Screenshot 2020-03-05 at 19.20.20.png | |
| Mar 5 2020, 7:29 PM |
Grafana dashboard: resourceloadermodule/startup/rate
It went from a stable 8 cache-miss backend requests per second (~480/min), to now 20-25 rps (~1300/min).
Close up (link):
(There is no moving average applied. Data points have a 15min resolution window from Graphite)
This suggests it started 2020-02-05 between 15:00-15:30, and graduallly tripped over the hour that followed.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| ATS: unset client req Accept-Encoding on ats-be | operations/puppet | production | +17 -12 |
Indeed. I suspect that maybe it is now getting both gzip and no-gzip traffic and/or no longer sharing CDN cache between the two. It's meant to internally only do no-gzip to the app layer though.
I suspect this is due to the fact that we are unsetting Accept-Encoding in do_global_send_request, which happens right before sending the request to the origin. We are thus unsetting the server request header. However, the check related to proxy.config.http.cache.ignore_accept_encoding_mismatch uses the client request field, which we leave untouched in Lua. It might thus be enough to perform the unsetting in do_global_read_request instead.
Change 577551 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: unset client req Accept-Encoding on ats-be
Theory confirmed in labs:
# cache hit without Accept-Encoding ema@traffic-cache-atstext-buster:~$ curl -v -H "X-Forwarded-Proto: https" -H "Host: en.wikipedia.beta.wmflabs.org" '127.0.0.1:3128/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector' 2>&1 | grep X-Cache < X-Cache-Int: traffic-cache-atstext-buster hit # cache misses with Accept-Encoding ema@traffic-cache-atstext-buster:~$ curl -v -H "Accept-Encoding: gzip" -H "X-Forwarded-Proto: https" -H "Host: en.wikipedia.beta.wmflabs.org" '127.0.0.1:3128/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector' 2>&1 | grep X-Cache < X-Cache-Int: traffic-cache-atstext-buster miss ema@traffic-cache-atstext-buster:~$ curl -v -H "Accept-Encoding: gzip" -H "X-Forwarded-Proto: https" -H "Host: en.wikipedia.beta.wmflabs.org" '127.0.0.1:3128/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector' 2>&1 | grep X-Cache < X-Cache-Int: traffic-cache-atstext-buster miss
Also confirmed that the proposed patch https://gerrit.wikimedia.org/r/577551 fixes the problem:
# cache hit with or without AE ema@traffic-cache-atstext-buster:~$ curl -v -H "Accept-Encoding: gzip" -H "X-Forwarded-Proto: https" -H "Host: en.wikipedia.beta.wmflabs.org" '127.0.0.1:3128/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector' 2>&1 | grep X-Cache < X-Cache-Int: traffic-cache-atstext-buster hit ema@traffic-cache-atstext-buster:~$ curl -v -H "X-Forwarded-Proto: https" -H "Host: en.wikipedia.beta.wmflabs.org" '127.0.0.1:3128/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector' 2>&1 | grep X-Cache < X-Cache-Int: traffic-cache-atstext-buster hit
Change 577551 merged by Ema:
[operations/puppet@production] ATS: unset client req Accept-Encoding on ats-be
