Page MenuHomePhabricator

Intermittent transcode failure 'An unknown error occurred in storage backend "local-swift-codfw".'
Open, Needs TriagePublic

Description

Seen a few errors like this during transcodes:

An unknown error occurred in storage backend "local-swift-codfw".

Only a few over the past, probably just intermittent oddities that need a retry.

Event Timeline

Yes, it looks like the upload server (or some other instance) can't properly process files above a file size of 500-600 MB. It doesn't matter if it's an image or a video. For me, there was always an error with larger file sizes in the last months, so the error should be easily reproducible with a simple larger file. Factors like dimensions or pixel count don't matter here, image sizes up to 65535x65535 (4.3 gigapixels: largest dimensions PNG supports), work.

If I remember correctly, an upload of such file sizes used to work, but since a few months there seems to be a bug somehow.

Does uploading a larger file work for you or is this a general bug?

Thank you :)!

Upload fails with

Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.

Request from 2a01:cb15:xxxx via cp3056 cp3056, Varnish XID 110584618
Error: 503, Backend fetch failed at Tue, 05 Oct 2021 19:23:48 GMT upload by URL from IA: https://archive.org/details/da-capo-press-camera-notes-v-1-2-1897-1899

TheDJ subscribed.

@Yann that seems like a different error that should be filed as a separate ticket ?

Again upload failed for 714 MB PDF file from IA via upload-by-url. Error message: Request from 2a01:cb15:803d:5d00:39a8:3513:ce67:4196 via cp3062 cp3062, Varnish XID 212786691. Error: 503, Backend fetch failed at Wed, 06 Oct 2021 14:31:00 GMT

Source: https://archive.org/details/da-capo-press-camera-notes-v-3-4-1899-1901

@Yann that seems like a different error that should be filed as a separate ticket ?

Not sure. It seems a recurrent issue with large files,whatever the format.
See also https://commons.wikimedia.org/wiki/Commons_talk:Maximum_file_size#%3E_100_MB

https://archive.org/details/da-capo-press-camera-notes-v-3-4-1899-1901 also failed via chunked-upload protocol. Error message: 02579: FAILED: stashfailed: Internal error: Server failed to publish temporary file. This is easily reproductible, and it is becoming a serious issue.

Also from https://commons.wikimedia.org/wiki/Special:UploadStash , if I click on (view thumbnail), I get

Fetching thumbnail failed: Array ( [0] => Array ( [0] => http-timed-out [1] => http://thumbor.svc.eqiad.wmnet:8800/wikipedia/commons/thumb/temp/5/5e/20211006151957%21chunkedupload_721eb52ef6ed.pdf/page1-220px-20211006151957%21chunkedupload_721eb52ef6ed.pdf.jpg ) [1] => Array ( [0] => http-bad-status [1] => 0 [2] => Error ) ) URL = http://thumbor.svc.eqiad.wmnet:8800/wikipedia/commons/thumb/temp/5/5e/20211006151957%21chunkedupload_721eb52ef6ed.pdf/page1-220px-20211006151957%21chunkedupload_721eb52ef6ed.pdf.jpg

Just got this:

[urbanecm@mwmaint1002 ~/uploads]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=PantheraLeo1359531 .
Importing Files

Importing BurningShip_Wiki_x264_CRF4_20210820_4500p60_002.webm...failed. (An unknown error occurred in storage backend "local-swift-codfw".)

Found: 1
Failed: 1
[urbanecm@mwmaint1002 ~/uploads]$