⚓ T124101 Specific revisions of multiple files missing from Swift

		Status	Subtype	Assigned	Task
		Open		None	T124101 Specific revisions of multiple files missing from Swift - 404 Not Found returned
		Duplicate		None	T182822 Generate a list of files that are supposed to exist but 404s

Josve05a set Security to None.Jan 19 2016, 10:06 PM

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJan 19 2016, 10:06 PM

Bawolff added a subscriber: aaron.Jan 19 2016, 10:16 PM

Aklapper renamed this task from File not found (404 - The resource could not be found) to Specific revision of file on Commons triggers 404 (not found).Jan 20 2016, 1:36 PM

Josve05a added a project: User-Josve05a.Jan 20 2016, 7:15 PM

Josve05a moved this task from Backlog to Tasks to follow on the User-Josve05a board.

Steinsplitter moved this task from Incoming to Backlog on the Commons board.Jan 22 2016, 5:59 PM

matmarex edited projects, added SRE-swift-storage; removed Wikimedia-Media-storage, Multimedia, MediaWiki-File-management.Jan 26 2016, 3:54 PM

Poyekhali renamed this task from Specific revision of file on Commons triggers 404 (not found) to Specific revision of a file on Commons triggers 404 (not found).Apr 13 2016, 4:00 AM

Poyekhali triaged this task as High priority.

Poyekhali updated the task description. (Show Details)

Poyekhali subscribed.

Same with https://upload.wikimedia.org/wikipedia/en/archive/9/9a/20080218114104%21Internet_Download_Accelerator.png vs https://en.wikipedia.org/wiki/File:Internet_Download_Accelerator.png

404 Not Found

The resource could not be found.

File not found: /v1/AUTH_mw/wikipedia-en-local-public.9a/archive/9/9a/20060321114104%21Internet_Download_Accelerator.png

Josve05a updated the task description. (Show Details)May 21 2016, 1:08 AM

Josve05a moved this task from Tasks to follow to Continue to complain about on the User-Josve05a board.Jun 23 2016, 8:03 AM

Josve05a assigned this task to aaron.Jul 31 2016, 10:58 PM

aaron removed aaron as the assignee of this task.Aug 2 2016, 12:13 AM

aaron added a project: Multimedia.

And first revision of https://en.wikipedia.org/wiki/File:Parisian.PNG is also missing

https://upload.wikimedia.org/wikipedia/en/archive/6/68/20070717000058%21Parisian.PNG;

File not found: /v1/AUTH_mw/wikipedia-en-local-public.68/archive/6/68/20070717000058%21Parisian.PNG

Josve05a renamed this task from Specific revision of a file on Commons triggers 404 (not found) to Specific revision of a file triggers 404 (not found).Aug 11 2016, 7:03 PM

First revision of https://en.wikipedia.org/wiki/File:Greater_London_Authority_logo.png as well

https://upload.wikimedia.org/wikipedia/en/archive/7/78/20080828203602%21Greater_London_Authority_logo.png

File not found: /v1/AUTH_mw/wikipedia-en-local-public.78/archive/7/78/20080828203602%21Greater_London_Authority_logo.png

Josve05a added a project: SRE.Aug 11 2016, 8:43 PM

Josve05a renamed this task from Specific revision of a file triggers 404 (not found) to Specific revision of a files triggers 404 (not found).Aug 11 2016, 8:45 PM

Josve05a renamed this task from Specific revision of a files triggers 404 (not found) to Specific revisions of multiple files triggers 404 (not found).

Image https://commons.wikimedia.org/wiki/File:Cactaceae_(1082183341).jpg missing (there's only one revisions). File was uploaded 01:25, 7 June 2016

The 120px thumbnail is there, though.

Platonides renamed this task from Specific revisions of multiple files triggers 404 (not found) to Specific revisions of multiple files missing from Swift - 404 Not Found returned.Sep 16 2016, 4:46 PM

zhuyifei1999 subscribed.Nov 29 2016, 2:35 AM

This problems seems to increase. There was one thread on the the English VP today, and two on the German Forum.

Perhelion updated the task description. (Show Details)Nov 29 2016, 9:55 AM

Fae awarded a token.Nov 29 2016, 1:53 PM

Fae subscribed.

I don't see this as "high" priority, but I'm willing to be convinced otherwise. Since these are not current versions of files, it doesn't seem to affect most day-to-day work.

It doesn't seem like these files have anything in common. There are PNGs and JPGs, different file sizes, different resolutions, different upload dates...and different re-upload dates, too. There are even files from at least two different wikis.

Can anyone offer any insight into the similarities between these files?

This is a current version: https://commons.wikimedia.org/wiki/File:Burbuja_(1496994920).jpg

First revision of https://commons.wikimedia.org/wiki/File:J.J._Burns_NSRW1-0009.jpg as well

https://upload.wikimedia.org/wikipedia/commons/archive/b/b8/20071019085735%21J.J._Burns_NSRW1-0009.jpg

File not found: /v1/AUTH_mw/wikipedia-commons-local-public.b8/archive/b/b8/20071019085735%21J.J._Burns_NSRW1-0009.jpg

To be honest, I find priority "normal" to be worrying. If potentially unrecoverable data loss is not a high priority then what is?

@Srittau: Feel free to elaborate why this task is more urgent than other tasks on the project workboards. "Data loss" in general sounds like a severity categorization. Priority rather depends on how often and recent things happen I'd say...

MarkTraceur unsubscribed.Jan 27 2017, 10:16 PM

ArloJamesBarnes subscribed.Mar 31 2017, 12:07 AM

zhuyifei1999 moved this task from Backlog to File storage issues on the Commons board.Jul 25 2017, 10:25 AM

14:24 < Dragonfly6-7> https://commons.wikimedia.org/wiki/File:Burbuja_(1496994920).jpg does anyone else see this as broken?
14:25 < mutante> Dragonfly6-7: yea, confirmed. https://upload.wikimedia.org/wikipedia/commons/2/2e/Burbuja_%281496994920%29.jpg is 404

zhuyifei1999 added a subscriber: • DragonflySixtyseven.Dec 13 2017, 7:40 PM

Jojr149 removed a project: User-Josve05a.Feb 6 2018, 2:18 PM

@Jojr149: Please do not remove user projects from tasks.

• Niedzielski mentioned this in T187955: Page preview shows icon instead of thumbnail.Feb 21 2018, 10:49 PM

• Mholloway subscribed.Feb 21 2018, 10:51 PM

zhuyifei1999 mentioned this in T192253: First revision from 2006 of a specific file on Commons is missing.Apr 16 2018, 2:33 PM

Copied from my duplicate task:

https://commons.wikimedia.org/wiki/File:Accuracy_International_Arctic_Warfare_-_Psg_90.jpg

First revision is from 23:38, 1 December 2006 but the file does not exist. Current revision from 23:33, 3 November 2008 is fine though. (fine is a big word, the quality is shitty, but Wikimedia can't help that)

I tried 00 to ff for https://upload.wikimedia.org/wikipedia/commons/2/2e/Burbuja_%281496994920%29.jpg (guessing it might be misplaced) but it's just not there. https://commons.wikimedia.org/w/api.php?action=query&list=allimages&aisha1=5c898098c147202d918a63e2b3731b3eb9779051 does work, so the upload was fine.

Here is a list things, may not all be related to this issue.

https://commons.wikimedia.org/wiki/Commons:Deletion_requests/File:L%27%C3%A9glise_de_Sadirac.JPG
https://commons.wikimedia.org/wiki/Commons:Deletion_requests/File:Anne_Paugam,_directrice_g%C3%A9n%C3%A9rale_de_l%27AFD.jpg
https://commons.wikimedia.org/wiki/Commons:Deletion_requests/Image:Gauteng_eastrand_map.jpg

https://commons.wikimedia.org/wiki/Category:Files_with_404_errors
From this category: https://commons.wikimedia.org/wiki/File:Cimbriani_shield_pattern.svg (png thumbnail works, cached probably, svg missing)

Glitch?
https://commons.wikimedia.org/wiki/Commons:Deletion_requests/File:Long_Beach_Comic_%26_Horror_Con_2011_-_Aquamen_(6301708270).jpg

Possibly different case, "A handful of them got corrupted on upload." (but should that have resulted in 404?)
https://commons.wikimedia.org/wiki/Commons:Deletion_requests/File:Australian_Census_2011_demographic_map_-_New_South_Wales_by_POA_-_BCP_field_0147_Visitor_from_Different_SA2_in_Victoria_Age_0_14_years.svg

Dzahn unsubscribed.Jun 21 2018, 5:18 AM

https://commons.wikimedia.org/wiki/File:Chris_Benoit_in_the_Ring.jpg

7 previous revisions missing, maybe revdel worked like this in 2007?

Original of https://commons.wikimedia.org/wiki/File:ATX-Netzteil.jpg is missing too: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.e9/e/e9/ATX-Netzteil.jpg

In T124101#4538078, @Raymond wrote:

Original of https://commons.wikimedia.org/wiki/File:ATX-Netzteil.jpg is missing too: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.e9/e/e9/ATX-Netzteil.jpg

I've reuploaded it to https://commons.wikimedia.org/wiki/File:HEC_350W_ATX_power_supply.jpg because it was in use. Commons now reports it as duplicate. It remembers the hash, but the original file is gone.

The most suspicious development in this story is that these are Wikimedia users—some not even sysops on any wiki where a revision is missing—try to trace causes of the bug, whereas all people having shell access to Wikimedia servers are silent. Are files really missing from directories or only unreadable for some reason (permissions, file-system damage, I/O error… )? For files which are missing, how recently did they exist in the past? Which procedures (such as moving files from one storage to another) could potentially cause the data loss?

@Incnis_Mrsi at T198177 I also found some missing revisions. But the things that take me hours could be done in seconds with shell access.

In T124101#4789594, @AlexisJazz wrote:

But the things that take me hours could be done in seconds with shell access.

How do you think that would happen? If you have specific steps in mind, maybe we could expose those to power users. I can't think of any investigation that would be much more complicated without shell access though.

In T124101#4789680, @Tgr wrote:

In T124101#4789594, @AlexisJazz wrote:

But the things that take me hours could be done in seconds with shell access.

How do you think that would happen? If you have specific steps in mind, maybe we could expose those to power users. I can't think of any investigation that would be much more complicated without shell access though.

For T198177 I made hundreds of requests for filenames like https://upload.wikimedia.org/wikipedia/commons/archive/4/42/20180626085337%21WikipediaTomeRaider3.png using an educated guess for the approximate timestamp, then bruteforcing it.

Just typing "locate Clinton_power_station_1.jpg" and "locate ATX-Netzteil.jpg" would be way faster. (my revision finder didn't work for those anyway, but the files may still exist elsewhere.. I just don't know where.

Swift has filename prefix search but not substring search AFAIK.

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:29 PM

https://commons.wikimedia.org/wiki/File:President_Lula_and_Marisa.jpg first two revisions missing.

https://commons.wikimedia.org/wiki/File:Busan_tower_by_night.jpg First four revisions missing!

In T124101#2643954, @Platonides wrote:

Image https://commons.wikimedia.org/wiki/File:Cactaceae_(1082183341).jpg missing (there's only one revisions). File was uploaded 01:25, 7 June 2016

The 120px thumbnail is there, though.

The revision appears to be showing now?

(old comment that was waiting as draft in the browser)

Other urls found in the channel:
https://commons.wikimedia.org/wiki/File:Chirimena_(1341689114).jpg https://upload.wikimedia.org/wikipedia/commons/a/ab/Chirimena_%281341689114%29.jpg
https://commons.wikimedia.org/wiki/File:Después_de_la_llovizna_(1082176465).jpg https://upload.wikimedia.org/wikipedia/commons/8/85/Despu%C3%A9s_de_la_llovizna_%281082176465%29.jpg
https://commons.wikimedia.org/wiki/File:Humboldt_(2119142521).jpg https://upload.wikimedia.org/wikipedia/commons/3/3b/Humboldt_%282119142521%29.jpg

The file was reuploaded by Denniss on September 2016, but the original file is still missing, TheSandDoctor.

https://en.wikipedia.org/wiki/File:Wilderness_Society_in_front_of_Kindness_House.jpg

Only one revision and it's gone. (I'll reupload it though)

"Upload error

The file "mwstore://local-multiwrite/local-public/b/b2/Wilderness_Society_in_front_of_Kindness_House.jpg" is in an inconsistent state within the internal storage backends"

I guess I'm not reuploading anything!

Wtf: Internet archive archived the file on April 2, 2021. Yesterday! And now it's gone? How?

Krinkle edited projects, added Structured Data Engineering; removed Multimedia.Apr 3 2021, 2:57 AM

Restricted Application added a project: Structured-Data-Backlog. · View Herald TranscriptApr 3 2021, 2:57 AM

Krinkle added a project: MediaWiki-File-management.Apr 3 2021, 2:57 AM

CBogen moved this task from Triage to Multimedia on the Structured-Data-Backlog board.Apr 5 2021, 4:12 PM

https://en.wikipedia.org/wiki/File:Roosevelt_High_School_(St._Louis).jpg first revision missing https://upload.wikimedia.org/wikipedia/en/archive/2/2f/20080118004233%21Roosevelt_High_School_%28St._Louis%29.jpg

https://en.wikipedia.org/wiki/File:Logo_of_the_International_Practical_Shooting_Confederation.png
One revision, uploaded 17:17, 23 August 2021 aaaand it's gone.

https://upload.wikimedia.org/wikipedia/en/d/de/Logo_of_the_International_Practical_Shooting_Confederation.png stated: File not found: /v1/AUTH_mw/wikipedia-en-local-public.de/d/de/Logo_of_the_International_Practical_Shooting_Confederation.png

Now this just links to my overwrite. There is no thumbnail for the initial revision, no link. It is reported as 288 × 346 (54 KB) but it's nowhere to be seen.

FlightTime moved the file on 06:35, 27 August 2021‎. Before that it was working.

@MarkTraceur @CBogen @Tgr can someone investigate https://upload.wikimedia.org/wikipedia/en/d/de/Logo_of_the_International_Practical_Shooting_Confederation.png ? This one's interesting because it's both recent and the only revision (before my overwrite) went missing.

• Suran38 awarded a token.Aug 28 2021, 2:07 PM

• Suran38 subscribed.

@MarkTraceur @Tgr Could someone with shell access at least look in some of the relevant archive directories and see if anything is in there. Maybe the filenames have been changed or corrupted or it's an encoding issue.

https://upload.wikimedia.org/wikipedia/en/archive/2/2f/20080118004233%21Roosevelt_High_School_%28St._Louis%29.jpg
/v1/AUTH_mw/wikipedia-en-local-public.2f/archive/2/2f/20080118004233%21Roosevelt_High_School_%28St._Louis%29.jpg

https://upload.wikimedia.org/wikipedia/commons/archive/0/06/20070422000344%21President_Lula_and_Marisa.jpg
/v1/AUTH_mw/wikipedia-commons-local-public.06/archive/0/06/20070422000344%21President_Lula_and_Marisa.jpg

These aren't directories, they are Swift containers. Our documentation on how to do common filesystem-like operations on Swift is not great. In theory it seems like calling FileBackend::getFileList() should work.

tgr@mwmaint1002:~$ mwscript shell.php commonswiki
Psy Shell v0.11.5 (PHP 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf5 — cli) by Justin Hileman

>>> $srv = MediaWiki\MediaWikiServices::getInstance()
=> MediaWiki\MediaWikiServices {#89}

>>> $fb = $srv->getFileBackendGroup()->get( 'local-swift-eqiad' )
=> SwiftFileBackend {#3685}

>>> $fb->getFileList( [ 'dir' => 'mwstore://local-swift-eqiad/wikipedia-commons-local-public.06/archive/0/06' ] );
=> null

I guess that wasn't the right way to do it. (Wouldn't it be super nice if we used those mwstore:// urls consistently in error messages?)

mwstore://local-swift-eqiad/local-public/archive/0/06 at least returns an iterator, but it's empty. One could probably trace the logic in thumb.php to find the right set of commands to turn a HTTP URL into an mwstore:// URL, but I don't have the time right now.

In T124101#7314869, @AlexisJazz wrote:

@MarkTraceur @CBogen @Tgr can someone investigate https://upload.wikimedia.org/wikipedia/en/d/de/Logo_of_the_International_Practical_Shooting_Confederation.png ? This one's interesting because it's both recent and the only revision (before my overwrite) went missing.

Seemingly I didn't link the file page? Here: https://en.wikipedia.org/wiki/File:Logo_of_the_International_Practical_Shooting_Confederation.png

The initial revision by Pbrks from 17:17, 23 August 2021 still has a revert link, so it's not deleted, but there's no thumbnail. The revert link is https://en.wikipedia.org/w/index.php?title=File:Logo_of_the_International_Practical_Shooting_Confederation.png&action=revert&oldimage= which strikes me as odd.

Listings look empty:

>>> $be = MediaWiki\MediaWikiServices::getInstance()->getFileBackendGroup()->get( 'local-multiwrite' );

>>> $iter = $be->getFileList( [ 'dir' => 'mwstore://local-multiwrite/local-public/archive/2/2f' ] );
=> SwiftFileBackendFileList {#6191}

>>> foreach ( $iter as $f ) { if ( str_starts_with( $f, '20080118004233' ) ) echo "$f\n"; }
>>> $iter = $be->getFileList( [ 'dir' => 'mwstore://local-multiwrite/local-public/archive/0/06' ] );
=> SwiftFileBackendFileList {#6278}

>>> foreach ( $iter as $f ) { if ( str_starts_with( $f, '20070422000344' ) ) echo "$f\n"; }

If there were entries, they'd show like:

>>> foreach ( $iter as $f ) { if ( str_starts_with( $f, '200704' ) ) echo "$f\n"; }
20070425123203!Flag_map_of_Tunisia.svg

El_Grafo subscribed.Mar 15 2023, 10:54 AM

Specific revisions of multiple files missing from Swift - 404 Not Found returned
Open, MediumPublic
Actions

Description

Related Objects
Search...

Event Timeline

	Josve05a
	Jan 19 2016, 10:05 PM

Specific revisions of multiple files missing from Swift - 404 Not Found returnedOpen, MediumPublicActions

Description

Related ObjectsSearch...

Event Timeline

Specific revisions of multiple files missing from Swift - 404 Not Found returned
Open, MediumPublic
Actions

Related Objects
Search...