apt-staging: add headers to prevent CDN caching
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fnegri
	Aug 19 2025, 11:02 AM

Description

The HTTP server at https://apt-staging.wikimedia.org/ is fronted by our CDN cache, and that can serve stale content. We should change the Nginx settings in apt-staging2001 to add some HTTP headers that set a maximum cache time.

For example, I added a new package to apt-staging following this wiki page. The package was visible at https://apt-staging.wikimedia.org/wikimedia-staging/pool/main/w/wikireplicas-utils/, but other files like https://apt-staging.wikimedia.org/wikimedia-staging/dists/bookworm-wikimedia/main/binary-amd64/Packages.gz and https://apt-staging.wikimedia.org/wikimedia-staging/dists/bookworm-wikimedia/InRelease were showing a cached old version, not including the new package.

As a consequence, running reprepro checkupdate in apt1002 could not find and import the new package.

Workaround: purging the cache following this guide fixes the issue but must be done for each affected URL.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	cache::text: set apt-staging to NOT cache	operations/puppet	production	+2 -0

Customize query in gerrit

Event Timeline

fnegri created this task.Aug 19 2025, 11:02 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2025, 11:02 AM

Change #1180234 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] cache::text: set apt-staging to NOT cache

https://gerrit.wikimedia.org/r/1180234

gerritbot added a project: Patch-For-Review.Aug 19 2025, 9:37 PM

How about https://gerrit.wikimedia.org/r/c/operations/puppet/+/1180234/1/hieradata/role/common/cache/text.yaml to just turn off the caching at the CDN while keeping the other benefits of the CDN?

@Dzahn fine with me, but if there's an easy way to keep e.g. a 5-minute cache it could be nice to have. I'll let @Joe have the final word.

MoritzMuehlenhoff edited projects, added Traffic; removed Infrastructure-Foundations.Aug 20 2025, 2:48 PM

In T402284#11101087, @fnegri wrote:

@Dzahn fine with me, but if there's an easy way to keep e.g. a 5-minute cache it could be nice to have. I'll let @Joe have the final word.

You can add headers on the applayer to have them served by the CDN and set a preferred caching time (let us know if you need help with that). @Dzahn's patch also works if you want to skip the CDN layer but still get the benefits of it, so that's another way. It depends on what you want and what is acceptable for apt-staging so let's work from there.

In T402284#11120837, @ssingh wrote:

In T402284#11101087, @fnegri wrote:

@Dzahn fine with me, but if there's an easy way to keep e.g. a 5-minute cache it could be nice to have. I'll let @Joe have the final word.

You can add headers on the applayer to have them served by the CDN and set a preferred caching time (let us know if you need help with that). @Dzahn's patch also works if you want to skip the CDN layer but still get the benefits of it, so that's another way. It depends on what you want and what is acceptable for apt-staging so let's work from there.