Page MenuHomePhabricator

The file "XXX" is in an inconsistent state within the internal storage backends
Open, MediumPublic

Description

When trying to process T289781: Server side upload for PantheraLeo1359531, I see the following error:

[urbanecm@mwmaint1002 ~/T289781]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user='PantheraLeo1359531' .
Importing Files

Importing BurningShip_Wiki_x264_CRF4_20210820_4500p60_002.webm...failed. (An unknown error occurred in storage backend "local-swift-codfw".)

Found: 1
Failed: 1
[urbanecm@mwmaint1002 ~/T289781]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user='PantheraLeo1359531' .
Importing Files

Importing BurningShip_Wiki_x264_CRF4_20210820_4500p60_002.webm...failed. (The file "mwstore://local-multiwrite/local-public/f/fd/BurningShip_Wiki_x264_CRF4_20210820_4500p60_002.webm" is in an inconsistent
state within the internal storage backends)

Found: 1
Failed: 1
[urbanecm@mwmaint1002 ~/T289781]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user='PantheraLeo1359531' .
Importing Files

Importing BurningShip_Wiki_x264_CRF4_20210820_4500p60_002.webm...failed. (The file "mwstore://local-multiwrite/local-public/f/fd/BurningShip_Wiki_x264_CRF4_20210820_4500p60_002.webm" is in an inconsistent
state within the internal storage backends)

Found: 1
Failed: 1
[urbanecm@mwmaint1002 ~/T289781]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user='PantheraLeo1359531' .
Importing Files

Importing BurningShip_Wiki_x264_CRF4_20210820_4500p60_002.webm...failed. (The file "mwstore://local-multiwrite/local-public/f/fd/BurningShip_Wiki_x264_CRF4_20210820_4500p60_002.webm" is in an inconsistent
state within the internal storage backends)

Found: 1
Failed: 1
[urbanecm@mwmaint1002 ~/T289781]$

As you can see, I tried to process the request several times, failing all the times.

This also applies to T290722, see:

[urbanecm@mwmaint1002 ~/uploads2]$ mwscript importImages.php --wiki=commonswiki --sleep=30 --comment-ext=txt --user='Nikola Smolenski' .
Importing Files

Importing Policiski recnik 1.pdf...done.
Importing Policiski recnik 2.pdf...done.
Importing Policiski recnik 3.pdf...failed. (An unknown error occurred in storage backend "local-swift-codfw".)

Found: 3
Added: 2
Failed: 1
[urbanecm@mwmaint1002 ~/uploads2]$ mwscript importImages.php --wiki=commonswiki --sleep=30 --comment-ext=txt --user='Nikola Smolenski' .
Importing Files

Policiski recnik 1.pdf exists, skipping
Policiski recnik 2.pdf exists, skipping
Importing Policiski recnik 3.pdf...failed. (The file "mwstore://local-multiwrite/local-public/c/c4/Policiski_recnik_3.pdf" is in an inconsistent state within the internal storage backends)

Found: 3
Skipped: 2
Failed: 1
[urbanecm@mwmaint1002 ~/uploads2]$ mwscript importImages.php --wiki=commonswiki --sleep=30 --comment-ext=txt --user='Nikola Smolenski' .
Importing Files

Policiski recnik 1.pdf exists, skipping
Policiski recnik 2.pdf exists, skipping
Importing Policiski recnik 3.pdf...failed. (The file "mwstore://local-multiwrite/local-public/c/c4/Policiski_recnik_3.pdf" is in an inconsistent state within the internal storage backends)

Found: 3
Skipped: 2
Failed: 1
[urbanecm@mwmaint1002 ~/uploads2]$

I'm unable to figure out what's wrong -- turning over to you for investigation.

Event Timeline

If it is of any help, when I try to open the file with Adobe's acroread, it reports "stat buffer overflow" error, while evince opens it nicely (if slowly).

Marostegui triaged this task as Medium priority.Sep 20 2021, 5:10 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-23T15:45:16Z] <urbanecm> Start server-side upload for 1 video file (T289781), testing whether T291137 is still an issue

Mentioned in SAL (#wikimedia-operations) [2021-10-23T15:45:16Z] <urbanecm> Start server-side upload for 1 video file (T289781), testing whether T291137 is still an issue

It still was an issue.

Can someone investigate please? This is pending here for more than a month, while it's an issue that breaks a certain part of MW (with no workaround available). In another words, this looks like a major severity issue to me :)).

The original video file is gone; is there still an issue with the PDF collection? Thumbor has had problems handling PDFs (cf T337649), alternatively it might be that the offending PDF is malformed in some way (thumbor is pickier about files fitting their spec than some desktop tools, not unreasonably).
I'm afraid we don't really have enough visibility through the upload stack to be able to investigate this sort of problem very well (since obviously uploads in general are working OK).
@Frostly for future reference, tasks already tagged sre-swift-storage needn't be referred to the clinic duty person (whose role is more about ensuring unassigned tasks get passed to the relevant team).