Page MenuHomePhabricator

ChartMuseum responses are cached in the CDN with default (24h) ttl
Closed, ResolvedPublic

Description

Today @jeena hit a snag when updating a chart: the new version 0.0.2 didn't show up on ChartMuseum until I manually purged the URL from the CDN, despite it having been over half an hour since the patch was merged.

Using mwscript purgeList.php to manually purge the URL in question fixed things.

The CDN definitely caches the responses:

✔️ cdanis@localhost ~ 🕒☕ curl -sIXGET https://helm-charts.wikimedia.org/stable/api/charts/rdf-streaming-updater
HTTP/2 200 
age: 88
x-cache: cp1081 miss, cp1087 hit/2
[...]

From the CDN's POV, ChartMuseum doesn't send a Cache-Control header at all:

1✔️ cdanis@chartmuseum1001.eqiad.wmnet ~ 🕒☕ curl -sIXGET $(RESOLVE https://helm-charts.wikimedia.org/stable/api/charts/rdf-streaming-updater localhost)
2HTTP/1.1 200 OK
3content-type: application/json; charset=utf-8
4x-request-id: 1872110a-8b8b-44eb-bed0-2cdf21a31d46
5date: Thu, 21 Jan 2021 19:53:57 GMT
6content-length: 1046
7x-envoy-upstream-service-time: 126
8server: envoy

so I believe these all get cached for the default_ttl of 24h. That seems far too long.

I think ChartMuseum should be configured to send a Cache-Control response header that specifies s-maxage of approx 1 to 5 minutes.

Related Objects

Event Timeline

hieradata/role/common/cache/text.yaml has:

60   helm-charts.wikimedia.org:
61     caching: 'normal'

That should confirm that it is indeed the 24 hour default.

An easy way to do this would be to just switch 'normal' to 'pass' here. Then there would be no caching at all. We do the same for other services like cxserver, API, blubberoid, config-master, debmonitor and others.

Would that be ok? If not then it probably needs a new type of caching besides the existing: normal, pass, pipe and websockets?

An easy way to do this would be to just switch 'normal' to 'pass' here. Then there would be no caching at all. We do the same for other services like cxserver, API, blubberoid, config-master, debmonitor and others.

Would that be ok? If not then it probably needs a new type of caching besides the existing: normal, pass, pipe and websockets?

That's a workable option for sure, but I wanted to ask about setting a short s-maxage first.

While it's unlikely to ever be an actual problem, we expose a lot of important internal services with caching: pass. It simplifies configurations, but creates other risk.

Unfortunately, upstream was not very responsive on my question about adding Cache-Control (https://github.com/helm/chartmuseum/issues/368). I wonder why this problem arised only recently as we ran this configuration from the start.

As ChartMuseum does an internal caching of the repo objects as well, I think we can switch to 'pass' for now. Will prepare a patch.

Change 657772 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] cache_text: Disable caching for helm-charts.wikimedia.org

https://gerrit.wikimedia.org/r/657772

Change 657772 merged by JMeybohm:
[operations/puppet@production] cache_text: Disable caching for helm-charts.wikimedia.org

https://gerrit.wikimedia.org/r/657772

Unfortunately, upstream was not very responsive on my question about adding Cache-Control (https://github.com/helm/chartmuseum/issues/368). I wonder why this problem arised only recently as we ran this configuration from the start.

As ChartMuseum does an internal caching of the repo objects as well, I think we can switch to 'pass' for now. Will prepare a patch.

Too bad about upstream.

'pass' for now sounds good.

If in the future you do want to cache in the CDN, looks like Envoy can inject response headers (after a migration to v3 config):
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#custom-request-response-headers
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route.proto#envoy-v3-api-field-config-route-v3-routeconfiguration-response-headers-to-add

Closing this as cache is disabled now.