FileBackendStore::ingestFreshFileStats: Could not stat file
Closed, InvalidPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	TheresNoTime
	Oct 11 2023, 8:03 PM

Description

Error

mwversion: 1.41.0-wmf.30
reqId: 5584ed45-3567-4494-b7bc-daf0273f420a
Find reqId in Logstash

normalized_message

FileBackendStore::ingestFreshFileStats: Could not stat file mwstore://local-swift-eqiad/local-public/archive/f/f4/20231011194328!Armijski_General_KJUG_-_Armija.png

exception.trace

Impact

Notes

20:56:47 <Kizule> Hi, is there some maintenance or something like that? Commons is throwing me "Failed to commit operations" when I'm using FileImporter for moving files from Serbian Wikipedia to Commons?

Details

Request URL: https://commons.wikimedia.org/wiki/Special:ImportFile

	Subject	Repo	Branch	Lines +/-
	Log and show reasons when commit operations fail	mediawiki/extensions/FileImporter	master	+21 -19
	filebackend: Fix and add missing @return documentation	mediawiki/core	master	+5 -3

Customize query in gerrit

Related Objects

Mentioned In: T348937: Some or all of the undeletion failed
T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg
T348690: "Failed to commit operations" on Wikimedia Commons while using FileImporter

Event Timeline

TheresNoTime created this task.Oct 11 2023, 8:03 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 11 2023, 8:03 PM

TheresNoTime mentioned this in T348690: "Failed to commit operations" on Wikimedia Commons while using FileImporter.Oct 11 2023, 8:32 PM

I think that Move-Files-To-Commons is good/correct project to add, since this is happening with FileImporter.

Now it didn't work with just one image revision and 3 history revisions. I'm positive that this isn't happening only to me.

https://sr.wikipedia.org/wiki/Датотека:Православна_црква_у_Чуругу.jpg

Kizule mentioned this in T348586: File not found: /v1/AUTH_mw/wikipedia-commons-local-public.7e/7/7e/EC02-0162-69_l_%2824374651802%29.jpg.Oct 15 2023, 2:14 AM

TheresNoTime mentioned this in T348937: Some or all of the undeletion failed.Oct 15 2023, 9:40 PM

thiemowmde subscribed.Oct 16 2023, 7:03 AM

Change 966145 had a related patch set uploaded (by Thiemo Kreuz (WMDE); author: Thiemo Kreuz (WMDE)):

[mediawiki/core@master] filebackend: Fix and add missing @return documentation

https://gerrit.wikimedia.org/r/966145

gerritbot added a project: Patch-For-Review.Oct 16 2023, 9:52 AM

Digging one step deeper I find the reason for the "Could not stat file" is a HTTP 401 (Unauthorized) in 'SwiftFileBackend::doGetFileStatMulti' for the HEAD request done in this method. We see a sequence of such 401 in the log. I believe this is because there is one for each version of the file (3 for the 3 versions we see on https://commons.wikimedia.org/wiki/File:Armijski_General_KJUG_-_Armija.png).

Why this 401 happens I can't tell.

The original file is already deleted: https://sr.wikipedia.org/wiki/%D0%94%D0%B0%D1%82%D0%BE%D1%82%D0%B5%D0%BA%D0%B0:Armijski_General_KJUG_-_Armija.png?action=history. It would be great to know when it was originally created, i.e. in which year.

The original file is already deleted: https://sr.wikipedia.org/wiki/%D0%94%D0%B0%D1%82%D0%BE%D1%82%D0%B5%D0%BA%D0%B0:Armijski_General_KJUG_-_Armija.png?action=history. It would be great to know when it was originally created, i.e. in which year.

The first two versions of the file were uploaded in January 2014 and the third in October 2019.

Ok, thanks! (I just realized I can see the original dates in the version history on Commons. That's the whole point of FileImporter, isn't it?) 😅️

My suspicion was that the file is so old that either one of the files on disc or one of the database entries got corrupted, along with many other uploads at the time. But this doesn't seem to be the case. The corruptions I think of happened about 10 years earlier.

In T348688#9253849, @JJMC89 wrote:

The original file is already deleted: https://sr.wikipedia.org/wiki/%D0%94%D0%B0%D1%82%D0%BE%D1%82%D0%B5%D0%BA%D0%B0:Armijski_General_KJUG_-_Armija.png?action=history. It would be great to know when it was originally created, i.e. in which year.

The first two versions of the file were uploaded in January 2014 and the third in October 2019.

It's right.

In T348688#9253133, @thiemowmde wrote:

Digging one step deeper I find the reason for the "Could not stat file" is a HTTP 401 (Unauthorized) in 'SwiftFileBackend::doGetFileStatMulti' for the HEAD request done in this method. We see a sequence of such 401 in the log. I believe this is because there is one for each version of the file (3 for the 3 versions we see on https://commons.wikimedia.org/wiki/File:Armijski_General_KJUG_-_Armija.png).

Why this 401 happens I can't tell.

The original file is already deleted: https://sr.wikipedia.org/wiki/%D0%94%D0%B0%D1%82%D0%BE%D1%82%D0%B5%D0%BA%D0%B0:Armijski_General_KJUG_-_Armija.png?action=history. It would be great to know when it was originally created, i.e. in which year.

I deleted it because I managed to transfer it after a few tries, but this is still happening in case there are multiple file revisions.

thiemowmde added projects: SRE-swift-storage, WMDE-TechWish-Maintenance.Oct 17 2023, 7:55 AM

thiemowmde added subscribers: awight, WMDE-Fisch.

Change 966145 merged by jenkins-bot:

[mediawiki/core@master] filebackend: Fix and add missing @return documentation

https://gerrit.wikimedia.org/r/966145

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 6:31 PM

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.2; 2023-10-24).Oct 18 2023, 7:00 PM

Hi, can someone check logs for "https://commons.wikimedia.org/wiki/File:Divizijski_General_KJUG_-_Armija.png"?

I tried again, import failed with "Failed to commit operations." again, and only the latest revision of the image (from 2019) was imported, not and revisions from 2014.

https://sr.wikipedia.org/wiki/%D0%94%D0%B0%D1%82%D0%BE%D1%82%D0%B5%D0%BA%D0%B0:Divizijski_General_KJUG_-_Armija.png

Kizule moved this task from Untriaged to October 2023 on the Wikimedia-production-error board.Oct 21 2023, 6:12 AM

@Kizule if you want logs looked at, could you specify what date/time (preferably in UTC) you mean, please? That file (I think you meant https://commons.wikimedia.org/wiki/File:Divizijski_General_KJUG_-_Armija.png without the embedded quote at the end) was deleted shortly before your comment...

In T348688#9272055, @MatthewVernon wrote:

@Kizule if you want logs looked at, could you specify what date/time (preferably in UTC) you mean, please? That file (I think you meant https://commons.wikimedia.org/wiki/File:Divizijski_General_KJUG_-_Armija.png without the embedded quote at the end) was deleted shortly before your comment...

About the same time and date when I've posted the comment here. Maybe few minutes before.

I find four PUTs around that time period -

moss-fe2001.codfw.wmnet: Oct 21 06:08:26 moss-fe2001 proxy-server: 10.192.0.75 10.192.32.51 21/Oct/2023/06/08/26 PUT /v1/AUTH_mw/wikipedia-commons-local-public.97/9/97/Divizijski_General_KJUG_-_Armija.png HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk0521e7420... 36332 - e42ed163fbcf822f4876693a4450f8b3 txa3277d9a925242fe93801-0065336ada - 0.0444 - - 1697868506.277877808 1697868506.322266579 0
ms-fe1009.eqiad.wmnet: Oct 21 06:08:26 ms-fe1009 proxy-server: 10.192.0.75 10.64.0.166 21/Oct/2023/06/08/26 PUT /v1/AUTH_mw/wikipedia-commons-local-public.97/9/97/Divizijski_General_KJUG_-_Armija.png HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk4f6415c17... 36332 - e42ed163fbcf822f4876693a4450f8b3 tx102a59fc3de549e8bc692-0065336ada - 0.0932 - - 1697868506.420666695 1697868506.513847589 0
ms-fe1010.eqiad.wmnet: Oct 21 06:08:39 ms-fe1010 proxy-server: 10.67.135.191 10.64.16.92 21/Oct/2023/06/08/39 PUT /v1/AUTH_mw/wikipedia-commons-local-thumb.97/9/97/Divizijski_General_KJUG_-_Armija.png/116px-Divizijski_General_KJUG_-_Armija.png HTTP/1.0 201 - python-swiftclient-4.0.0 AUTH_tk946ab2cc0... 14948 - - tx2c19cd2ddf6e482ea9b09-0065336ae7 - 0.0464 - - 1697868519.822117805 1697868519.868529081 0
ms-fe1012.eqiad.wmnet: Oct 21 06:35:58 ms-fe1012 proxy-server: 10.67.153.120 10.64.130.2 21/Oct/2023/06/35/58 PUT /v1/AUTH_mw/wikipedia-commons-local-thumb.97/9/97/Divizijski_General_KJUG_-_Armija.png/116px-Divizijski_General_KJUG_-_Armija.png HTTP/1.0 201 - python-swiftclient-4.0.0 AUTH_tk946ab2cc0... 14948 - - tx81b3d079c88148db88860-006533714e - 0.0320 - - 1697870158.898442507 1697870158.930422068 0

So that's an upload of one image to both clusters (as expected) and then a thumbnail likewise to both clusters.

Change 793481 had a related patch set uploaded (by Thiemo Kreuz (WMDE); author: Thiemo Kreuz (WMDE)):

[mediawiki/extensions/FileImporter@master] Log and show reasons when commit operations fail

https://gerrit.wikimedia.org/r/793481

gerritbot added a project: Patch-For-Review.Oct 26 2023, 9:14 AM

Change 793481 merged by jenkins-bot: