Page MenuHomePhabricator

FileOperation error "SwiftFileBackend::addMissingMetadata: {path} was not stored with SHA-1 metadata."
Closed, ResolvedPublic

Description

Error

Request ID: W5mZegrAIFcAAFZSjTwAAAAD

normalized message
SwiftFileBackend::addMissingMetadata: {path} was not stored with SHA-1 metadata.
example
SwiftFileBackend::addMissingMetadata: mwstore://local-swift-codfw/local-thumb/0/05/Wes_Craven_2010.jpg/140px-Wes_Craven_2010.jpg was not stored with SHA-1 metadata.

channel: FileOperation
level: ERROR

http_method: GET
wiki: commons.wikimedia.org
url: /w/thumb.php?f=Wes_Craven_2010.jpg&w=140

Notes

Past reports that might be related:

Event Timeline

Krinkle created this task.Sep 12 2018, 11:16 PM
Restricted Application added projects: Commons, Multimedia. · View Herald TranscriptSep 12 2018, 11:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gilles assigned this task to aaron.Sep 13 2018, 4:44 PM
Gilles added subscribers: aaron, Gilles.

This probably happens for thumbnails generated with thumbor and then accessed via thumb.php. What's the point of that custom x-object-meta-sha1base36 header for thumbnails, though? The code doesn't say. It could be something that's only really useful for originals and thumbnails just happen to get the same treatment.

@aaron can probably shed some light on this when he's back from his vacation. Assigning the task to him to provide some context on that header and whether it's necessary for MediaWiki to bother populating it for thumbnails (in which case thumbor should generate the same thing).

aaron added a comment.Oct 1 2018, 9:34 PM

It's used for originals. I don't think it matters much for thumbnails, but it's hard to cleanly tell that to SwiftFileBackend. It seems like it might be easiest to have thumbor hash the local file and save the metadata in the PUT request to avoid these errors (and slowness of triggering a GET to POST the missing data).

Gilles added a comment.Oct 1 2018, 9:55 PM

Well, that's a lot of storage space to waste saving useless metadata on thumbnail objects, imho. If it's not needed, we shouldn't store it. Surely we can add an option to SwiftFileBackend or something for the thumb case. I can look into it, I just wanted confirmation that this was only useful for originals.

Gilles claimed this task.Oct 1 2018, 9:56 PM
aaron added a comment.Oct 2 2018, 11:47 PM

Is it that much space? If you add an option, you have to have getFileStat return some dummy value for the SHA1 and also not have that mess up the logic in doOperations(), which is why it seemed easier to just include the header.

aaron added a comment.Oct 2 2018, 11:50 PM

If you go that route, then something like having the getFileSha1() and the sha1 field being null for certain containers plus having doOperations and friends pass a flag to getFileStat/Sha1 to have the current behavior of lazy-loading and not using null, it might work.

Gilles added a comment.Nov 8 2018, 3:35 PM

Header name + data is 55 bytes longs. We have 1.2 billion thumbnails in Swift. That's 66 GiB of data. Which represents 0.02% of our Swift storage space. Not earth-shattering savings, I'll grant you that, but I think we should get rid of data we don't need. It does affect speed a little as well when fetching thumbnails from Swift.

Change 472608 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] filebackend: avoiding computing file SHA-1 hashes unless needed

https://gerrit.wikimedia.org/r/472608

Gilles reassigned this task from Gilles to aaron.Nov 9 2018, 7:56 AM
Gilles triaged this task as Low priority.

Change 472608 merged by jenkins-bot:
[mediawiki/core@master] filebackend: avoiding computing file SHA-1 hashes unless needed

https://gerrit.wikimedia.org/r/472608

aaron closed this task as Resolved.Dec 11 2018, 7:59 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Wed, Aug 28, 11:09 PM