Page MenuHomePhabricator

Swift sends ETAG without double-quotes
Open, LowPublic

Description

Hosted picture have an ETAG without double quote like:

$ curl -I https://upload.wikimedia.org/wikipedia/commons/thumb/6/69/Flag_of_South_Carolina.svg/23px-Flag_of_South_Carolina.svg.png | grep etag
etag: dfb0c6f55f2d606b33afc130f05c7f88

But a few other (static?) images have an etag with double quotes:

$ curl -I https://bm.wikipedia.org/static/images/project-logos/bmwiki.png | grep etag
etag: "3e7c-5a6b0464619c2"

It should be coherent, and it seems the norm says it has to have double-quote, see https://developer.mozilla.org/en/docs/Web/HTTP/Headers/ETag

The reason for the discrepancy is that the former image is hosted on Swift, which sends unquoted ETags: https://bugs.launchpad.net/swift/+bug/1099087. The bug is fixed in Swift 2.24.0.

Event Timeline

Kelson created this task.Jun 24 2020, 10:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 24 2020, 10:09 AM
ema triaged this task as Medium priority.Jun 24 2020, 10:20 AM
ema added a project: Traffic.
Restricted Application added a project: Operations. · View Herald TranscriptJun 24 2020, 10:20 AM
ema moved this task from Triage to Caching on the Traffic board.Jul 2 2020, 11:21 AM

As it turns out, this is due to a deliberate bug in Swift: https://bugs.launchpad.net/swift/+bug/1099087

When swift was originally written, it was to replace an existing system. The requirements for swift were, "must be better than the old system and customers can't notice". That meant we had to port over a couple of the bugs from the old system (like no quotes on the etag).

Openstack/swift 2.24.0 fixes the issue (or removes the feature, depending on how you want to see it). We are currently running 2.10.2, and I'm not sure if an upgrade is in any way on our roadmap. @fgiunchedi?

ema renamed this task from ETAG response headers not always with double-quotes to Swift sends ETAG without double-quotes.Tue, Jul 21, 12:07 PM
ema edited projects, added SRE-swift-storage; removed serviceops.
ema updated the task description. (Show Details)

As it turns out, this is due to a deliberate bug in Swift: https://bugs.launchpad.net/swift/+bug/1099087

When swift was originally written, it was to replace an existing system. The requirements for swift were, "must be better than the old system and customers can't notice". That meant we had to port over a couple of the bugs from the old system (like no quotes on the etag).

Openstack/swift 2.24.0 fixes the issue (or removes the feature, depending on how you want to see it). We are currently running 2.10.2, and I'm not sure if an upgrade is in any way on our roadmap. @fgiunchedi?

Yes we'll be upgrading Swift as part of the regular Debian release upgrade. Specifically this means we'll be moving to Swift 2.19 for Buster and >= 2.25 for Bullseye

ema added a comment.Tue, Jul 21, 12:34 PM

Openstack/swift 2.24.0 fixes the issue (or removes the feature, depending on how you want to see it). We are currently running 2.10.2, and I'm not sure if an upgrade is in any way on our roadmap. @fgiunchedi?

Yes we'll be upgrading Swift as part of the regular Debian release upgrade. Specifically this means we'll be moving to Swift 2.19 for Buster and >= 2.25 for Bullseye

Alright, so it's on the radar but not in the immediate future. The issue seems to be mostly about following RFC 7232, which is definitely something we want to do, but not a functional problem when it comes to our CDN internals. I see Varnish sending conditional requests to ATS with If-None-Match and ATS responding appropriately with 304/200 for objects with and without quotes in ETag. @Kelson: have you experienced a functional issue on Kiwix/openZIM due to this?

@ema not really this is case which had to be handled in MWoffliner. This is all.

ema lowered the priority of this task from Medium to Low.Tue, Jul 21, 1:43 PM

@Kelson: alright, this will be solved with the upgrade to Bullseye then. Thank you!