Right now https://commons.wikimedia.org/wiki/File:Fawiki500k_celebration_by_Behdad_Abedi_(180).jpg doesn't show anything and when you click on "Original file" it gives 404 "File not found: /v1/AUTH_mw/wikipedia-commons-local-public.e6/e/e6/Fawiki500k_celebration_by_Behdad_Abedi_%28180%29.jpg"
Description
Description
Related Objects
Related Objects
- Mentioned In
- T161918: videoscalers (mw1168, mw1169) - high load / overheating
T161476: Commons File:Assemblea_Costituente_1946_(2).svg missing after file move
T161943: Getting intermittent "inconsistent state within the internal storage backends" exceptions
T161916: could not load image from commons - Mentioned Here
- T64057: Image move on Commons messed up: No file description page and only file
T162609: Swift version and distro upgrade
T161916: could not load image from commons
T161910: wikicommons image became invicible
T153565: MediaWiki file operations are fragile, causing occasional data loss
T111838: Some files had disappeared from Commons after renaming
T160640: Rack and Setup ms-be1028-ms-1039
T161864: 404 error while accessing some images files e.g. djvu and jpg
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
Comment Actions
Random debugging:
hphpd> $f = wfFindFile(Title::newFromText('File:Fawiki500k_celebration_by_Behdad_Abedi_(180).jpg'))
$f = wfFindFile(Title::newFromText('File:Fawiki500k_celebration_by_Behdad_Abedi_(180).jpg'))
hphpd> =$f->exists();
=$f->exists();
true
hphpd> =$f->repo->backend->fileExists( ['src' => $f->repo->resolveToStoragePath( $f->getVirtualUrl() ) ]);
=$f->repo->backend->fileExists( ['src' => $f->repo->resolveToStoragePath( $f->getVirtualUrl() ) ]);
falseVs a known good file:
$f = wfFindFile(Title::newFromText('File:Voltairine_de_Cleyre_(Age_35).jpg'))
hphpd> =$f->exists();
=$f->exists();
true
hphpd> =$f->repo->backend->fileExists( ['src' => $f->repo->resolveToStoragePath( $f->getVirtualUrl() ) ]);
=$f->repo->backend->fileExists( ['src' => $f->repo->resolveToStoragePath( $f->getVirtualUrl() ) ]);
trueSo at a minimum, swift certainly thinks the file doesn't exist, while the mediawiki database thinks it does.
Comment Actions
More cases brought up:
- https://commons.wikimedia.org/wiki/File:Fawiki500k_celebration_by_Behdad_Abedi_(180).jpg mentioned in https://commons.wikimedia.org/wiki/Commons:Village_pump#Sudden_file_corruption.2C_long_after_uploading
- https://de.wikipedia.org/wiki/Datei:Taiwan.JPG and https://de.wikipedia.org/wiki/Datei:Mayr_Andreas.jpg mentioned in https://de.wikipedia.org/wiki/Wikipedia:Fragen_zur_Wikipedia#Defekte_Bilder
Comment Actions
@Ladsgroup not related to thumbor no since thumbor isn't in production yet (though "thumbor in production" is a Q4 goal now)
The timeline lines up with the current swift in eqiad expansion going on in T160640, we have experienced files disappearing in the past (e.g. in T111838) though related to files moves.
At the time of T111838 a script was published in https://gerrit.wikimedia.org/r/#/c/249494/ to find files present in filebackend but not in mediawiki, I believe we'd need a similar script to do an audit the other way, find all files in mediawiki missing in swift
Comment Actions
T153565: MediaWiki file operations are fragile, causing occasional data loss, you might find a lot :(
Comment Actions
Indeed it is one of the long-standing bugs in SRE-swift-storage and likely the interaction between mediawiki and swift :(
Comment Actions
I can successfully see https://commons.wikimedia.org/wiki/File:Fawiki500k_celebration_by_Behdad_Abedi_(180).jpg now, ditto for https://commons.wikimedia.org/wiki/File:PL_J%C3%B3zef_Ignacy_Kraszewski-Poezye_tom_2.djvu but not https://commons.wikimedia.org/wiki/File:Wykolejony_%28Gruszecki%29_24.jpg yet so it might be a sign of swift converging.
I'm assuming the first two were not re-uploaded again (not seeing any new uploads in the File: history) ?
I'm still looking into why the files 404'd from swift's point of view, as a rebalance/expansion shouldn't be causing files to disappear of course
Comment Actions
Since some files linked here seem to 200 now (instead of 404) I'm lowering to "high", I'll keep looking at what might be causing this during a rebalance
Comment Actions
The book https://commons.wikimedia.org/wiki/File:PL_J%C3%B3zef_Ignacy_Kraszewski-Poezye_tom_2.djvu is transcribed at Polish Wikisource: we have many pages without scans now...
The same with https://pl.wikisource.org/wiki/Indeks:Andrzej_Kijowski_-_Listopadowy_wiecz%C3%B3r.djvu
Can you upload these files from archives?
Comment Actions
Hi, I'll add another one: https://commons.wikimedia.org/wiki/File:MyanmarChin.png
https://upload.wikimedia.org/wikipedia/commons/9/93/MyanmarChin.png
The text 404 is File not found: /v1/AUTH_mw/wikipedia-commons-local-public.93/9/93/MyanmarChin.png (maybe is missing a slash between public and 93, instead a point?)
Comment Actions
Possibly another related case reported by a different editor: https://commons.wikimedia.org/wiki/File:Yaroslava_Shvedova.JPG, mentioned at https://commons.wikimedia.org/wiki/Commons:Administrators%27_noticeboard#File:Yaroslava_Shvedova.JPG.
Comment Actions
Setting back to UBN!, due to the number of duplicate tasks and that there may be more files affected other than those reported here. We cannot afford to wait more files to be lost unexpectedly.
Comment Actions
This seems to be a gradual process. When I commented on https://commons.wikimedia.org/wiki/File:Yaroslava_Shvedova.JPG at COM:AN, I could still access the thumbnail in the file history. Now this is gone too.
Update: on the other hand, https://commons.wikimedia.org/wiki/File:Wykolejony_%28Gruszecki%29_24.jpg is back again, but its upload log is missing.
Comment Actions
All pages from this book have empty upload log, so it is rather nothing strange here.
The file was renamed; the upload log is available under its previous name: https://commons.wikimedia.org/w/index.php?title=Special:Log&page=File%3AWykolejony013+a24.jpg.
Also, already came back:
https://commons.wikimedia.org/wiki/File:Andrzej_Kijowski_-_Listopadowy_wiecz%C3%B3r.djvu
https://de.wikipedia.org/wiki/Datei:Taiwan.JPG
https://de.wikipedia.org/wiki/Datei:Mayr_Andreas.jpg https://commons.wikimedia.org/wiki/File:MyanmarChin.png
Reported previously and not accessible at the moment:
https://commons.wikimedia.org/wiki/File:50_%D0%B4%D0%BE%D0%BC._%D0%A3%D0%BB%D0%B8%D1%86%D0%B0_%D0%9D%D0%B5%D0%BA%D1%80%D0%B0%D1%81%D0%BE%D0%B2%D0%B0._%D0%93%D0%BE%D1%80%D0%BE%D0%B4_%D0%A1%D0%B5%D0%B2%D0%B5%D1%80%D0%BE%D0%B4%D0%B2%D0%B8%D0%BD%D1%81%D0%BA._%D0%A4%D0%BE%D1%82%D0%BE_%D0%90%D0%BB%D0%B5%D0%BA%D1%81%D0%B5%D1%8F_%D0%A9%D0%B5%D0%BA%D0%B8%D0%BD%D0%BE%D0%B2%D0%B0.jpg
https://commons.wikimedia.org/wiki/File:Vladimir_Frolochkin.JPG
https://commons.wikimedia.org/wiki/File:School_Gyrls_at_Paramount_Studios.jpg
https://commons.wikimedia.org/wiki/File:Yaroslava_Shvedova.JPG
Files reported earlier have already appeared again. The ones reported later are still inaccessible.
Note, that if no special action has been taken concerning the files that are already available, it may mean that random files are still disappearing for some (quite long) period of time and then appearing again.
Comment Actions
@Revent reported these urls
(the video works but scrolling to bottom and re transcoding them fails)
Comment Actions
The video 'works' because when you simply play it, you view some transcode based on your preferences. If you try to view the original video by clicking the link directly under the thumbnail, you get a 404.
Comment Actions
The current list of affected files (at least, of ones with a failed transcode that make them apparent) is....
https://commons.wikimedia.org/wiki/File:X5Flare_AIA193.webm
https://commons.wikimedia.org/wiki/File:Wagejot-4804259.webm
https://commons.wikimedia.org/wiki/File:Walking_Keage_Incline.webm
https://commons.wikimedia.org/wiki/File:T%C5%AFn%C4%9B_pod_mal%C3%BDm_vodop%C3%A1dem_Peri%C4%8Dn%C3%ADk_a_koup%C3%A1n%C3%AD.webm
https://commons.wikimedia.org/wiki/File:%D0%A1%D0%92-%D0%94%D0%9D%D0%A0-564._%D0%9D%D0%BE%D0%B2%D1%8B%D0%B9_%D0%B3%D0%BE%D0%B4_%D0%B2_%D0%94%D0%BE%D0%BD%D0%B5%D1%86%D0%BA%D0%B5.webm
This file has re-appeared in the last half-hour or so...
https://commons.wikimedia.org/wiki/File:Waterfront-PlymouthMA.webm
Comment Actions
Of the ones I listed before (videos), these are now back...
https://commons.wikimedia.org/wiki/File:X5Flare_AIA193.webm
https://commons.wikimedia.org/wiki/File:Wagejot-4804259.webm
https://commons.wikimedia.org/wiki/File:T%C5%AFn%C4%9B_pod_mal%C3%BDm_vodop%C3%A1dem_Peri%C4%8Dn%C3%ADk_a_koup%C3%A1n%C3%AD.webm
https://commons.wikimedia.org/wiki/File:%D0%A1%D0%92-%D0%94%D0%9D%D0%A0-564._%D0%9D%D0%BE%D0%B2%D1%8B%D0%B9_%D0%B3%D0%BE%D0%B4_%D0%B2_%D0%94%D0%BE%D0%BD%D0%B5%D1%86%D0%BA%D0%B5.webm
This one is still AWOL...
https://commons.wikimedia.org/wiki/File:Walking_Keage_Incline.webm
Comment Actions
Also available already:
https://commons.wikimedia.org/wiki/File:Vladimir_Frolochkin.JPG
https://commons.wikimedia.org/wiki/File:School_Gyrls_at_Paramount_Studios.jpg
This one was moved/deleted in the meattime and deleted image is not available (not recoverable):
https://commons.wikimedia.org/wiki/Special:Undelete/File:50_%D0%B4%D0%BE%D0%BC._%D0%A3%D0%BB%D0%B8%D1%86%D0%B0_%D0%9D%D0%B5%D0%BA%D1%80%D0%B0%D1%81%D0%BE%D0%B2%D0%B0._%D0%93%D0%BE%D1%80%D0%BE%D0%B4_%D0%A1%D0%B5%D0%B2%D0%B5%D1%80%D0%BE%D0%B4%D0%B2%D0%B8%D0%BD%D1%81%D0%BA._%D0%A4%D0%BE%D1%82%D0%BE_%D0%90%D0%BB%D0%B5%D0%BA%D1%81%D0%B5%D1%8F_%D0%A9%D0%B5%D0%BA%D0%B8%D0%BD%D0%BE%D0%B2%D0%B0.jpg
Comment Actions
https://commons.wikimedia.org/wiki/File:Walking_Keage_Incline.webm reappeared, and has been reset
Comment Actions
It seems this is no longer affecting our files, so I lowered the priority. Is this issue still occuring?
Comment Actions
I think all disappearing files should be back now as rebalance has finished. We are working on bringing swift all to the same version in T162609: Swift version and distro upgrade which should be completed in the next few days, after that I'll issue another rebalance which shouldn't have the same side effects reported here.
Comment Actions
We are rebalancing both swift clusters but haven't seen a reoccurence of this (namely files disappearing and then reappearing. Tentatively closing but please reopen if this happens again.