As it turns out, this is due to a deliberate bug in Swift: https://bugs.launchpad.net/swift/+bug/1099087

When swift was originally written, it was to replace an existing system. The requirements for swift were, "must be better than the old system and customers can't notice". That meant we had to port over a couple of the bugs from the old system (like no quotes on the etag).

Openstack/swift 2.24.0 fixes the issue (or removes the feature, depending on how you want to see it). We are currently running 2.10.2, and I'm not sure if an upgrade is in any way on our roadmap. @fgiunchedi?

In T256217#6322433, @ema wrote:

As it turns out, this is due to a deliberate bug in Swift: https://bugs.launchpad.net/swift/+bug/1099087

When swift was originally written, it was to replace an existing system. The requirements for swift were, "must be better than the old system and customers can't notice". That meant we had to port over a couple of the bugs from the old system (like no quotes on the etag).

Openstack/swift 2.24.0 fixes the issue (or removes the feature, depending on how you want to see it). We are currently running 2.10.2, and I'm not sure if an upgrade is in any way on our roadmap. @fgiunchedi?

Yes we'll be upgrading Swift as part of the regular Debian release upgrade. Specifically this means we'll be moving to Swift 2.19 for Buster and >= 2.25 for Bullseye

In T256217#6322461, @fgiunchedi wrote:

In T256217#6322433, @ema wrote:

Openstack/swift 2.24.0 fixes the issue (or removes the feature, depending on how you want to see it). We are currently running 2.10.2, and I'm not sure if an upgrade is in any way on our roadmap. @fgiunchedi?

Yes we'll be upgrading Swift as part of the regular Debian release upgrade. Specifically this means we'll be moving to Swift 2.19 for Buster and >= 2.25 for Bullseye

Alright, so it's on the radar but not in the immediate future. The issue seems to be mostly about following RFC 7232, which is definitely something we want to do, but not a functional problem when it comes to our CDN internals. I see Varnish sending conditional requests to ATS with If-None-Match and ATS responding appropriately with 304/200 for objects with and without quotes in ETag. @Kelson: have you experienced a functional issue on Kiwix/openZIM due to this?

@ema not really this is case which had to be handled in MWoffliner. This is all.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

I'm not sure since when, but based on us having <14 days ats-be storage, and based on there still beeing ETag headers on cached responses, I am guessing this is a pretty recent regression.

I'm finding that upload.wikimedia.org responses have neither ETag nor Last-Modified. This is also observed in T295556, but I'm skeptical of whether it is the same given the above caching, but perhaps these heades are both kept in Swift and still used post-Swift upgrade for pre-existing objects?

curl -I https://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Flag_of_South_Carolina.svg/23px-Flag_of_South_Carolina.svg.png
HTTP/2 200 
date: Thu, 26 May 2022 13:03:14 GMT
etag: 77423ce1b2c2a709de0db56d618d942c
server: ATS/8.0.8
content-type: image/png
content-length: 581
content-disposition: inline;filename*=UTF-8''Flag_of_South_Carolina.svg.png
last-modified: Wed, 20 Apr 2022 23:01:29 GMT
age: 6743
x-cache: cp3065 hit, cp3061 hit/190
x-cache-status: hit-front
server-timing: cache;desc="hit-front", host;desc="cp3061"
...
access-control-allow-origin: *
access-control-expose-headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache
timing-allow-origin: *
accept-ranges: bytes

Then, after purging the file page on Commons:

curl -I https://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Flag_of_South_Carolina.svg/23px-Flag_of_South_Carolina.svg.png
HTTP/2 200 
date: Thu, 26 May 2022 14:55:42 GMT
content-type: image/png
content-length: 581
content-disposition: inline;filename*=UTF-8''Flag_of_South_Carolina.svg.png
xkey: File:Flag_of_South_Carolina.svg
server: Thumbor/6.3.2
age: 3
x-cache: cp3065 hit, cp3061 hit/1
x-cache-status: hit-front
server-timing: cache;desc="hit-front", host;desc="cp3061"
..
access-control-allow-origin: *
access-control-expose-headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache
timing-allow-origin: *
accept-ranges: bytes

In T256217#7960730, @Krinkle wrote:

I'm not sure since when, but based on us having <14 days ats-be storage, and based on there still beeing ETag headers on cached responses, I am guessing this is a pretty recent regression.

I'm finding that upload.wikimedia.org responses have neither ETag nor Last-Modified. This is also observed in T295556, but I'm skeptical of whether it is the same given the above caching, but perhaps these heades are both kept in Swift and still used post-Swift upgrade for pre-existing objects?

I think that is just the difference between the file being generated & served by Thumbor (which doesn't seem to send ETag or LM headers), or by Swift from its object storage (which sends the non-RFC compliant ETag, and LM), note the Server header:

curl -I https://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Flag_of_South_Carolina.svg/23px-Flag_of_South_Carolina.svg.png
HTTP/2 200 
etag: 77423ce1b2c2a709de0db56d618d942c
server: ATS/8.0.8
last-modified: Wed, 20 Apr 2022 23:01:29 GMT
age: 6743
x-cache: cp3065 hit, cp3061 hit/190
[snip]

Then, after purging the file page on Commons:

curl -I https://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Flag_of_South_Carolina.svg/23px-Flag_of_South_Carolina.svg.png
HTTP/2 200 
server: Thumbor/6.3.2
age: 3
x-cache: cp3065 hit, cp3061 hit/1
[snip]

This appears to be configurable now in Swift 2.24.0 and later (we currently seem to be running 2.26.0 on 6/8 of frontends...), by enabling a piece of middleware and configuring RFC compliant ETag responses for specific Swift user accounts or containers:
https://docs.openstack.org/swift/latest/middleware.html#module-swift.common.middleware.etag_quoter

I can confirm that we're running a new-enough swift version everywhere that we could turn RFC-compliant ETags on. How confident are we that this won't break something?

@MatthewVernon That would be welcome on MWoffliner side! For the rest I can not say, but respecting the RFC sounds here quite important.

In T256217#9194798, @MatthewVernon wrote:

I can confirm that we're running a new-enough swift version everywhere that we could turn RFC-compliant ETags on. How confident are we that this won't break something?

I tested this locally on Swift 2.29. Even with the pipeline stage disabled, Swift is returning a 304 response when the INM header has a quoted ETag. It's stripping the quotes. Enabling this may lead to a temporary drop in 304 rate in ATS, but Swift will still give 304s either way, so the extra load should be tolerable.

Just enabling the pipeline stage doesn't do anything so is very unlikely to break anything. You have to set X-Account-Rfc-Compliant-Etags: true on the account for it to actually work:

swift post -H 'X-Account-Rfc-Compliant-Etags: true'

Then ETag response headers contain quotes. So roll it out, and if there's any problem, you can revert it at the account level without having to wait for puppet.