Page MenuHomePhabricator

Deleting file on Commons "Error deleting file: An unknown error occurred in storage backend "local-multiwrite"."
Closed, ResolvedPublic

Description

I'm trying to delete https://commons.wikimedia.org/wiki/File:Literature_II,_Harutyun_Surkhatian.djvu but the file won't delete, instead it gives the error.

Error deleting file: An unknown error occurred in storage backend "local-multiwrite".

The file looks to be corrupt in some way, perhaps uploading has failed and left us with a file description page and a corrupt djvu file (the file being non functional is why it has been nominated for deletion).

The original file, as uploaded https://upload.wikimedia.org/wikipedia/commons/d/df/Literature_II%2C_Harutyun_Surkhatian.djvu returns a 502 Bad Gateway error when you try to download it, so it's not possible to download and re-upload.

This issue appears to have been present for at least the last month, the file was tagged for speedy deletion, couldn't be deleted and somehow was converted into a regular DR, where it sat unclosed for three+ weeks because nobody could manage to delete the file.

T75094 may be a related bug, so I've set the same tags and subscribers, but opened a new ticket as the error isn't exactly the same and the file type is different.

Event Timeline

Nick updated the task description. (Show Details)

Is there an exception id or anything like that attached to the error? I can't find anything related to that in logstash ATM

Is there an exception id or anything like that attached to the error? I can't find anything related to that in logstash ATM

No, there's nothing but the message as it's copied above. I've done a quick screengrab in case it helps in anyway.

Screen Shot 2017-08-17 at 17.46.00.png (337×1 px, 71 KB)

The file is not accessible due to nginx on swift proxy machines yielding upstream sent too big header while reading response header from upstream and indeed x-content-dimensions is huge (~8kb) (see below)

IIRC we had foresee this and put protections in place to avoid generating huge x-content-dimensions for pathological cases like this, it could also be that nginx on swift proxy needs tuning to match said maximum length

X-Content-Dimensions: 2128x3445:1/2143x3454:2/2359x3662:3,581/2369x3668:4/2180x3429:5/2181x3430:6/2336x3647:7,41/2148x3409:8/2338x3649:9/2140x3405:10/2340x3649:11/2189x3435:12/2334x3646:13/2182x3431:14/2340x3650:15,27/2120x3328:16,18,36,42,50,68,104,106/2312x3632:17,31,49,59,547,549,551,583,593,611,613/2368x3680:19/2220x3407:20/2353x3658:21,573/2141x3342:22/2329x3643:23,99/2142x3342:24/2332x3645:25,69,545/2137x3339:26/2150x3347:28/2327x3642:29,35/2147x3345:30,54/2191x3373:32,80/2339x3649:33,73/2160x3353:34/2323x3639:37,565,591/2131x3335:38,58/2354x3658:39,585/2130x3334:40,76/2320x3637:43,569/2178x3365:44/2358x3661:45/2140x3340:46/2379x3674:47/2139x3340:48/2330x3643:51,57/2136x3338:52/2355x3659:53,83/2372x3670:55/2143x3342:56,98/2127x3332:60/2327x3641:61,561/2149x3346:62/2328x3642:63/2177x3364:64/2383x3677:65/2164x3356:66/2319x3637:67/2148x3346:70/2351x3657:71,587/2154x3350:72,74/2324x3640:75/2374x3671:77/2133x3337:78/2349x3656:79/2350x3656:81/2129x3334:82/2159x3353:84,96/2319x3636:85,97,543,567/2151x3348:86/2361x3663:87/2145x3344:88,102/2332x3644:89/2132x3336:90/2411x3694:91/2153x3349:92/2343x3652:93/2148x3345:94,110/2391x3681:95/2141x3341:100/2346x3653:101/2360x3662:103/2375x3672:105/2396x3685:107/2128x3333:108/2403x3689:109/2459x3686:111/2190x3353:112/2442x3675:113/2208x3365:114/2330x3638:115/2112x3344:116,118,120,146/2368x3662:117/2343x3647:119/2360x3657:121/2121x3350:122,184/2365x3661:123/2142x3363:124,154,186/2427x3699:125/2139x3361:126,204/2334x3641:127/2175x3383:128,176/2385x3673:129/2127x3354:130/2356x3655:131/2125x3352:132,136,142/2338x3643:133/2133x3357:134,152,198/2383x3672:135/2393x3678:137/2123x3351:138,190/2253x3643:139,143/2130x3356:140,148/2280x3659:141/2120x3349:144/2220x3623:145/2208x3616:147,149,151,157/2144x3364:150,172/2241x3636:153/2258x3646:155/2158x3373:156/2163x3376:158/2272x3616:159,169,179,185,205/2136x3359:160/2341x3659:161/2169x3380:162/2318x3645:163/2156x3372:164/2286x3625:165/2151x3368:166/2293x3629:167,209/2140x3361:168/2147x3366:170/2280x3621:171,177/2279x3620:173,197/2151x3369:174,210/2300x3634:175/2137x3360:178/2146x3366:180,182/2289x3627:181/2283x3623:183,187/2134x3358:188/2302x3635:189/2292x3628:191/2205x3402:192/2312x3641:193/2129x3354:194/2303x3636:195/2145x3364:196/2386x3687:199/2202x3400:200/2299x3633:201/2122x3350:202/2306x3637:203/2127x3353:206/2354x3667:207/2160x3374:208/2455x3676:211,213/2233x3365:212/2185x3334:214,226/2442x3668:215,237/2208x3349:216/2417x3652:217/2192x3338:218/2456x3676:219/2218x3355:220/2427x3658:221/2203x3346:222/2407x3646:223,269/2199x3343:224,232,254/2381x3629:225,249/2371x3623:227/2212x3351:228/2441x3667:229/2189x3337:230/2467x3684:231/2425x3657:233/2183x3333:234/2413x3649:235/2195x3340:236/2176x3328:238,244,246,252/2405x3644:239/2195x3341:240/2355x3612:241/2186x3335:242/2368x3620:243/2380x3628:245/2401x3642:247/2184x3333:248/2190x3337:250/2367x3620:251/2360x3615:253/2375x3625:255/2266x3386:256/2336x3600:257/2230x3363:258/2359x3615:259/2204x3346:260/2403x3643:261/2144x3328:262,268,282/2440x3667:263/2171x3345:264/2387x3633:265/2171x3346:266/2391x3635:267/2159x3338:270/2420x3654:271/2259x3401:272/2351x3609:273/2160x3338:274/2366x3620:275/2192x3358:276/2379x3607:277/2167x3343:278/2476x3670:279/2180x3351:280/2405x3624:281/2446x3651:283/2170x3345:284/2377x3632:285/2201x3334:286,310/2419x3658:287/2217x3344:288,582/2369x3627:289/2222x3347:290/2362x3622:291/2231x3354:292/2374x3630:293/2239x3359:294/2394x3642:295/2253x3368:296/2385x3637:297/2235x3356:298,300/2368x3626:299/2378x3632:301/2242x3360:302/2424x3661:303/2255x3369:304/2359x3620:305/2211x3340:306,544/2391x3640:307/2237x3357:308/2356x3618:309/2456x3708:311/2253x3439:312/2336x3632:313,367,389/2190x3400:314/2387x3665:315/2202x3407:316/2374x3657:317,381/2174x3390:318/2443x3700:319/2302x3469:320/2383x3662:321/2201x3407:322/2346x3639:323/2196x3404:324/2357x3645:325,349/2128x3328:326,332,336,344,350,354,360,378,380,384,386,412,620,642,650,662,664,670,672,678,688,692,696,698/2409x3678:327/2153x3344:328,638/2380x3660:329,387/2135x3333:330,334/2433x3693:331/2431x3692:333/2474x3719:335/2390x3667:337/2139x3335:338/2427x3690:339/2161x3349:340/2352x3642:341/2136x3333:342,356,406,410,674/2419x3685:343,413/2372x3655:345/2163x3350:346,654,668/2429x3691:347/2147x3340:348/2395x3670:351/2141x3337:352/2386x3664:353,377/2368x3652:355,403/2377x3658:357,369/2138x3334:358,660,694/2368x3653:359/2379x3659:361/2144x3339:362/2379x3660:363/2149x3341:364/2390x3666:365/2148x3340:366,632,634/2159x3348:368/2180x3361:370/2355x3644:371/2154x3344:372/2347x3639:373/2168x3353:374/2391x3667:375,385,395/2137x3334:376,652,682/2373x3656:379/2146x3340:382,636/2354x3644:383/2192x3369:388/2209x3379:390/2348x3640:391/2176x3358:392/2392x3668:393/2169x3354:394,624/2186x3365:396/2405x3676:397,405/2188x3366:398,686/2413x3681:399/2228x3391:400/2350x3641:401/2230x3392:402/2172x3356:404,408/2455x3707:407/2408x3678:409/2433x3694:411/2140x3336:414/2267x3474:415/2137x3324:416/2339x3591:417/2155x3360:418/2349x3597:419/2133x3347:420/2407x3634:421/2122x3339:422/2398x3628:423/2104x3328:424,452,454,458,474/2352x3599:425/2121x3339:426/2344x3594:427/2141x3351:428,488/2330x3585:429/2135x3347:430/2367x3608:431/2120x3338:432/2366x3607:433/2112x3333:434/2385x3620:435/2119x3337:436,456/2389x3622:437/2149x3357:438/2441x3655:439/2204x3390:440/2322x3580:441/2152x3358:442,478/2315x3575:443/2173x3371:444/2318x3577:445/2140x3351:446/2332x3586:447/2122x3340:448/2357x3602:449/2114x3334:450/2351x3598:451/2325x3581:453/2397x3628:455/2347x3596:457/2377x3489:459/2067x3306:460/2314x3574:461/2142x3352:462/2331x3586:463/2166x3367:464/2341x3592:465/2129x3344:466/2304x3568:467,483,485/2191x3383:468/2343x3593:469/2199x3387:470/2329x3584:471/2131x3345:472/2353x3600:473/2369x3610:475/2157x3361:476/2323x3580:477/2320x3578:479/2162x3365:480/2328x3583:481/2167x3368:482/2161x3363:484/2168x3344:486/2312x3573:487/2316x3576:489/2138x3349:490/2335x3620:491,509/2186x3379:492/2304x3600:493,495,505/2144x3353:494/2190x3382:496/2323x3612:497,521/2196x3385:498/2313x3606:499/2170x3369:500/2334x3619:501/2212x3366:502,510/2331x3617:503/2216x3369:504,508/2232x3379:506/2317x3608:507,517/2319x3609:511/2230x3439:512/2333x3618:513/2167x3401:514/2324x3613:515/2157x3395:516/2152x3391:518/2381x3649:519,539/2166x3400:520/2142x3305:522,536,540/2326x3614:523/2139x3303:524/2328x3615:525/2144x3307:526/2350x3629:527/2190x3336:528/2314x3607:529/2212x3350:530/2346x3627:531/2204x3345:532/2337x3621:533/2205x3345:534/2320x3610:535/2358x3634:537/2128x3296:538/2365x3666:541/2220x3347:542,576/2214x3343:546/2228x3352:548/2248x3364:550/2192x3328:552/2385x3678:553/2208x3339:554/2367x3667:555/2266x3376:556,562/2325x3640:557/2285x3388:558,592/2343x3651:559/2304x3401:560/2376x3672:563/2254x3369:564/2233x3355:566/2265x3376:568/2241x3360:570/2321x3638:571/2243x3361:572,588/2204x3336:574/2346x3654:575,607/2338x3463:577/2114x3343:578/2341x3650:579/2246x3363:580/2228x3351:584/2259x3372:586/2366x3666:589/2264x3375:590,610/2260x3372:594/2357x3661:595/2262x3374:596/2341x3651:597/2211x3341:598/2359x3661:599/2252x3367:600/2333x3645:601/2229x3352:602/2362x3664:603/2274x3381:604/2348x3655:605/2238x3358:606/2220x3346:608/2331x3644:609/2256x3370:612/2219x3346:614/2330x3563:615/2261x3373:616/2348x3575:617,649/2213x3342:618/2312x3552:619,623,627,633,651,653,661/2328x3562:621/2162x3349:622/2356x3581:625/2134x3332:626,646/2158x3347:628/2323x3559:629/2156x3346:630/2327x3562:631/2367x3588:635/2324x3560:637,643/2342x3571:639/2170x3355:640/2369x3589:641/2150x3342:644/2345x3573:645/2320x3557:647/2152x3343:648/2332x3565:655/2146x3339:656/2321x3558:657/2162x3350:658/2333x3566:659/2319x3557:663/2340x3570:665/2138x3335:666/2334x3598:667/2353x3610:669/2357x3613:671/2371x3622:673/2326x3593:675/2144x3338:676,684/2345x3605:677/2354x3611:679,691/2167x3353:680/2320x3589:681/2312x3584:683,689/2342x3603:685/2323x3591:687/2148x3341:690/2333x3598:693/2324x3592:695/2330x3596:697/2327x3594:699

I've downloaded the file Literature_II,_Harutyun_Surkhatian.djvu to check for corruption just in case @Nick though it might be valid and just pathological per-page dimensions.

https://people.wikimedia.org/~filippo/Literature_II_tom%252C_Harutyun_Surkhatian.djvu

We've abandoned X-Content-Dimensions, so I think we need to look at how we can clean it up. Is there a cheap way to find swift objects that have it set, or are we condemned to look at all originals?

I've downloaded the file Literature_II,_Harutyun_Surkhatian.djvu to check for corruption just in case @Nick though it might be valid and just pathological per-page dimensions.

https://people.wikimedia.org/~filippo/Literature_II_tom%252C_Harutyun_Surkhatian.djvu

It looks to be a corrupt file - tried a couple of djvu viewers and both throw a fit and crash when trying to open this file.

We've abandoned X-Content-Dimensions, so I think we need to look at how we can clean it up. Is there a cheap way to find swift objects that have it set, or are we condemned to look at all originals?

I think the latter :( The swift container listing API only returns selected headers for objects (e.g. date, content-type, etc).

The "good news" though is that we can walk the list of originals and unconditionally send X-Remove-Content-Dimensions: header (value doesn't matter) to remove the header without looking whether the original has the header or not, I'll open a followup task.

I've downloaded the file Literature_II,_Harutyun_Surkhatian.djvu to check for corruption just in case @Nick though it might be valid and just pathological per-page dimensions.

https://people.wikimedia.org/~filippo/Literature_II_tom%252C_Harutyun_Surkhatian.djvu

It looks to be a corrupt file - tried a couple of djvu viewers and both throw a fit and crash when trying to open this file.

Thanks! I've removed the problematic header manually now, the files should be deletable again!

And I guess we can do the cleanup "just" for file types that can be multipage (TIFF, DJVU, PDF), as the header size isn't problematic for single page documents.

@Nick could you try again to delete both files? thanks!

@Nick could you try again to delete both files? thanks!

@Nick: ping? ^

I have tried to delete the files several times in the past few days, but I cannot.

I have tried again and these two files still cannot be deleted.

@Jcb mind trying one more time? sorry for all the back and forth! I've purged the problematic header from both files and both datacenters now.

I could delete both files now. Thanks!

Should this task get closed as resolved?

Yes, I think the task is resolved now the two files are gone and I haven't seen any other involved files.

fgiunchedi claimed this task.

I agree let's resolve this, I'll followup on T175689: Remove X-Content-Dimensions for multipage originals on how to identify the files we couldn't clean up.

Please create a new task :)

Free free to mention this one in your report as it is similar indeed and might help.