Page MenuHomePhabricator
Create Task
Maniphest T164368

UploadWizard fails on beta with "Server failed to store temporary file" when uploading large files
Closed, ResolvedPublic
Actions

Description

Per Jean Fred:
I gave a try at uploading an Stl on the Beta cluster (a conversion from the Obj on Sketchfab) :)

(I had to resort to the old uploader as the UploadWizard kept failing with a "Cannot save temporary state" or smth like that)

(@MarkTraceur reproduced with DavidStatue.stl and confirmed the actual message "Internal error: Server failed to store temporary file.")

Related Objects

Event Timeline

• Elitre created this task.May 3 2017, 1:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 3 2017, 1:55 PM
• Elitre mentioned this in T158830: Communicate enabling Extension:3d on Commons.May 3 2017, 1:55 PM
MarkTraceur subscribed.May 3 2017, 2:00 PM

The actual error message (based on my upload of DavidStatue.stl) is "Internal error: Server failed to store temporary file."

• Elitre updated the task description. (Show Details)May 3 2017, 2:00 PM
MarkTraceur renamed this task from UploadWizard fails with "Cannot save temporary state" when uploading .stl file to UploadWizard fails with "Server failed to save temporary file" when uploading .stl file.May 3 2017, 2:01 PM
MarkTraceur updated the task description. (Show Details)
Keegan renamed this task from UploadWizard fails with "Server failed to save temporary file" when uploading .stl file to UploadWizard fails with "Server failed to store temporary file" when uploading .stl file.May 3 2017, 2:28 PM
matthiasmullie claimed this task.May 3 2017, 3:11 PM
matthiasmullie added a comment.May 4 2017, 12:14 PM

This is not only true for large .stl files. I also tried with a .jpg of similar size, and got the exact same error.

matthiasmullie renamed this task from UploadWizard fails with "Server failed to store temporary file" when uploading .stl file to UploadWizard fails on beta with "Server failed to store temporary file" when uploading large files.May 4 2017, 12:15 PM
matthiasmullie added a comment.May 4 2017, 1:29 PM

This is all I could find in the logs WRT to these failures: one for a large JPG file & one for a large STL file.
Issues seem identical, so this is probably not an issue with STL or 3D, but an issue with stashed files on beta.

JPG

FileOperation.log

2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation ERROR: HTTP 0 () in 'SwiftFileBackend::getAuthentication' (given '[]'): HTTP return code: 0  
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg.
  
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::preloadFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg.
  
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg.

runJobs.log

2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha runJobs ERROR: AssembleUploadChunks File:14uyf2tqn8x0.y1denb.1643.jpg filename=Zwevegem.jpg filekey=14uyf2tqn8x0.y1denb.1643.jpg session={"ip":"178.119.1.239","headers":"array(...)","sessionId":"fjsvmq4r5e3vafm7nh4bq3umuje548i6","userId":1643} requestId=WQskiwpEFhUAACovT5cAAAAE (uuid=347f1ffc8671420382f7c01f29583903,timestamp=1493902477) t=3072 error=UploadStashBadPathException: path doesn't exist

wfDebug.log

2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation ERROR: HTTP 0 () in 'SwiftFileBackend::getAuthentication' (given '[]'): HTTP return code: 0  
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg.
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::preloadFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg.
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha memcached DEBUG: getMulti(WANCache:v:filebackend:local-swift-eqiad:wikipedia-commons:file:d52c7a83d390559fdd6931f44c3132e38f3a1271)  
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg.
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha wfDebug DEBUG: UploadStash::getFile tried to get file at 14uyf2tqn8x0.y1denb.1643.jpg, but it doesn't exist {"seconds_elapsed":"3.1655","memory_used":"  2.5M"} 
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha DBQuery DEBUG: COMMIT {"method":"MWCallableUpdate::doUpdate","master":true,"runtime":0.0014629364013672} 
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha DBQuery DEBUG: COMMIT {"method":"Wikimedia\\Rdbms\\LoadBalancer::{closure}","master":false,"runtime":0.00068497657775879} 
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha runJobs ERROR: AssembleUploadChunks File:14uyf2tqn8x0.y1denb.1643.jpg filename=Zwevegem.jpg filekey=14uyf2tqn8x0.y1denb.1643.jpg session={"ip":"178.119.1.239","headers":"array(...)","sessionId":"fjsvmq4r5e3vafm7nh4bq3umuje548i6","userId":1643} requestId=WQskiwpEFhUAACovT5cAAAAE (uuid=347f1ffc8671420382f7c01f29583903,timestamp=1493902477) t=3072 error=UploadStashBadPathException: path doesn't exist  
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha wfDebug DEBUG: Request ended normally {"seconds_elapsed":"3.1789","memory_used":"  2.5M"} 
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha DBConnection DEBUG: Closing connection to database '10.68.18.35'.  
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha DBConnection DEBUG: Closing connection to database '10.68.23.30'.

STL

FileOperation.log

2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation ERROR: HTTP 0 () in 'SwiftFileBackend::getAuthentication' (given '[]'): HTTP return code: 0  
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er..
  
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::preloadFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er..
  
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er..

runJobs.log

2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha runJobs ERROR: AssembleUploadChunks File:14uyfcamvygg.dces4f.1643. filename=Test.stl filekey=14uyfcamvygg.dces4f.1643. session={"ip":"178.119.1.239","headers":"array(...)","sessionId":"fjsvmq4r5e3vafm7nh4bq3umuje548i6","userId":1643} requestId=WQslUQpEFhUAACWAaHQAAAAD (uuid=dfe359073ca146f49f0a46f49aaeddcd,timestamp=1493902675) t=3041 error=UploadStashBadPathException: path doesn't exist

wfDebug.log

2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation ERROR: HTTP 0 () in 'SwiftFileBackend::getAuthentication' (given '[]'): HTTP return code: 0  
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er..
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::preloadFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er..
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha memcached DEBUG: getMulti(WANCache:v:filebackend:local-swift-eqiad:wikipedia-commons:file:0acb022070650bbcc5446f9738b346f648ae7795)  
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er..
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha wfDebug DEBUG: UploadStash::getFile tried to get file at 14uyfcamvygg.dces4f.1643., but it doesn't exist {"seconds_elapsed":"3.1142","memory_used":"  2.5M"} 
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha DBQuery DEBUG: COMMIT {"method":"MWCallableUpdate::doUpdate","master":true,"runtime":0.00081706047058105} 
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha DBQuery DEBUG: COMMIT {"method":"Wikimedia\\Rdbms\\LoadBalancer::{closure}","master":false,"runtime":0.00062918663024902} 
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha runJobs ERROR: AssembleUploadChunks File:14uyfcamvygg.dces4f.1643. filename=Test.stl filekey=14uyfcamvygg.dces4f.1643. session={"ip":"178.119.1.239","headers":"array(...)","sessionId":"fjsvmq4r5e3vafm7nh4bq3umuje548i6","userId":1643} requestId=WQslUQpEFhUAACWAaHQAAAAD (uuid=dfe359073ca146f49f0a46f49aaeddcd,timestamp=1493902675) t=3041 error=UploadStashBadPathException: path doesn't exist  
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha wfDebug DEBUG: Request ended normally {"seconds_elapsed":"3.1254","memory_used":"  2.5M"} 
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha DBConnection DEBUG: Closing connection to database '10.68.18.35'.  
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha DBConnection DEBUG: Closing connection to database '10.68.23.30'.
matthiasmullie added a comment.May 4 2017, 1:51 PM

Grepping wfDebug.log for those temp filenames suggests that they were inserted to:

  • mwrepo://local/temp/2/22/20170504125236!P3HgMU.jpg
  • mwrepo://local/temp/b/b7/20170504125602!QYz3er.

but then stat was attempted on

  • mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg
  • mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er.

Not sure at this point what exactly these paths resolve to, so I don't know for sure if the difference in paths is the issue (and how that came to be)

JPG

2017-05-04 12:52:37 [WQskEwpEFhUAACWNdVIAAAAA] deployment-mediawiki05 commonswiki 1.30.0-alpha wfDebug DEBUG: UploadStash::stashFile inserting mwrepo://local/temp/2/22/20170504125236!P3HgMU.jpg under 14uyf2tqn8x0.y1denb.1643.jpg {"seconds_elapsed":"0.7946","memory_used":"  0.0M"} 
2017-05-04 12:52:37 [WQskEwpEFhUAACWNdVIAAAAA] deployment-mediawiki05 commonswiki 1.30.0-alpha DBQuery DEBUG: INSERT  INTO `uploadstash` (us_id,us_user,us_key,us_orig_path,us_path,us_props,us_size,us_sha1,us_mime,us_media_type,us_image_width,us_image_height,us_image_bits,us_source_type,us_timestamp,us_status) VALUES (NULL,'1643','14uyf2tqn8x0.y1denb.1643.jpg','/tmp/P3HgMU','mwrepo://local/temp/2/22/20170504125236!P3HgMU.jpg','a:12:{s:5:\"width\";i:10481;s:6:\"height\";i:3081;s:4:\"bits\";i:8;s:10:\"fileExists\";b:1;s:4:\"size\";i:5242880;s:9:\"file-mime\";s:10:\"image/jpeg\";s:10:\"major_mime\";s:5:\"image\";s:10:\"minor_mime\";s:4:\"jpeg\";s:4:\"mime\";s:10:\"image/jpeg\";s:4:\"sha1\";s:31:\"8w8ynmopxel5j7x80ojbzc1uimcdrna\";s:8:\"metadata\";s:1311:\"a:43:{s:4:\"Make\";s:9:\"Panasonic\";s:5:\"Model\";s:8:\"DMC-GX80\";s:11:\"Orientation\";i:1;s:11:\"XResolution\";s:5:\"180/1\";s:11:\"YResolution\";s:5:\"180/1\";s:14:\"ResolutionUnit\";i:2;s:8:\"Software\";s:7:\"Ver.1.1\";s:8:\"DateTime\";s:19:\"2017:02:02 08:12:16\";s:12:\"ExposureTime\";s:5:\"1/320\";s:7:\"FNumber\";s:4:\"11/5\";s:15:\"ExposureProgram\";i:2;s:15:\"ISOSpeedRatings\";i:200;s:11:\"ExifVersion\";s:4:\"0230\";s:16:\"DateTimeOriginal\";s:19:\"2017:02:02 08:12:16\";s:23:\"ComponentsConfiguration\";a:5:{i:0;i:1;i:1;i:2;i:2;i:3;i:3;i:0;s:5:\"_type\";s:2:\"ol\";}s:22:\"CompressedBitsPerPixel\";s:3:\"2/1\";s:17:\"ExposureBiasValue\";s:3:\"0/1\";s:16:\"MaxApertureValue\";s:9:\"8747/5713\";s:12:\"MeteringMode\";i:5;s:11:\"LightSource\";i:0;s:5:\"Flash\";i:16;s:11:\"FocalLength\";s:4:\"43/1\";s:10:\"ColorSpace\";i:1;s:13:\"SensingMethod\";i:2;s:10:\"FileSource\";i:3;s:9:\"SceneType\";i:1;s:12:\"ExposureMode\";i:0;s:12:\"WhiteBalance\";i:0;s:16:\"DigitalZoomRatio\";s:3:\"0/1\";s:21:\"FocalLengthIn35mmFilm\";i:86;s:16:\"SceneCaptureType\";i:0;s:11:\"GainControl\";i:1;s:8:\"Contrast\";i:0;s:10:\"Saturation\";i:0;s:9:\"Sharpness\";i:0;s:4:\"Lens\";s:19:\"LUMIX G 42.5mm F1.7\";s:17:\"DateTimeDigitized\";s:19:\"2017:02:02 08:12:16\";s:15:\"SublocationDest\";s:0:\"\";s:8:\"CityDest\";s:0:\"\";s:11:\"CountryDest\";s:0:\"\";s:19:\"ProvinceOrStateDest\";s:0:\"\";s:10:\"iimVersion\";i:4;s:22:\"MEDIAWIKI_EXIF_VERSION\";i:2;}\";s:10:\"media_type\";s:6:\"BITMAP\";}','5242880','8w8ynmopxel5j7x80ojbzc1uimcdrna','image/jpeg','BITMAP','10481','3081','8','file','20170504125237','finished') {"method":"UploadStash::stashFile","master":true,"runtime":0.0077919960021973} 
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg.
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::preloadFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg.
2017-05-04 12:54:41 [WQskiwpEFhUAACovT5cAAAAE] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/2/22/20170504125236!P3HgMU.jpg.

STL

2017-05-04 12:56:03 [WQsk4QpEFhUAACV9VcMAAAAX] deployment-mediawiki05 commonswiki 1.30.0-alpha wfDebug DEBUG: UploadStash::stashFile inserting mwrepo://local/temp/b/b7/20170504125602!QYz3er. under 14uyfcamvygg.dces4f.1643. {"seconds_elapsed":"0.9266","memory_used":"  0.0M"} 
2017-05-04 12:56:03 [WQsk4QpEFhUAACV9VcMAAAAX] deployment-mediawiki05 commonswiki 1.30.0-alpha DBQuery DEBUG: INSERT  INTO `uploadstash` (us_id,us_user,us_key,us_orig_path,us_path,us_props,us_size,us_sha1,us_mime,us_media_type,us_image_width,us_image_height,us_image_bits,us_source_type,us_timestamp,us_status) VALUES (NULL,'1643','14uyfcamvygg.dces4f.1643.','/tmp/QYz3er','mwrepo://local/temp/b/b7/20170504125602!QYz3er.','a:12:{s:10:\"fileExists\";b:1;s:4:\"size\";i:5242880;s:9:\"file-mime\";s:15:\"unknown/unknown\";s:10:\"major_mime\";s:8:\"chemical\";s:10:\"minor_mime\";s:12:\"x-mdl-rgfile\";s:4:\"mime\";s:21:\"chemical/x-mdl-rgfile\";s:4:\"sha1\";s:31:\"tmkhsib1tzv1h09x35nejf1ldcy2gvw\";s:8:\"metadata\";s:0:\"\";s:5:\"width\";i:0;s:6:\"height\";i:0;s:4:\"bits\";i:0;s:10:\"media_type\";s:7:\"DRAWING\";}','5242880','tmkhsib1tzv1h09x35nejf1ldcy2gvw','chemical/x-mdl-rgfile','DRAWING','0','0','0','file','20170504125603','finished') {"method":"UploadStash::stashFile","master":true,"runtime":0.00078296661376953} 
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er..
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::preloadFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er..
2017-05-04 12:57:59 [WQslUQpEFhUAACWAaHQAAAAD] deployment-jobrunner02 commonswiki 1.30.0-alpha FileOperation WARNING: FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-eqiad/local-temp/b/b7/20170504125602!QYz3er..
Fuzheado subscribed.May 6 2017, 12:45 PM
Fuzheado added a comment.May 6 2017, 12:51 PM

FYI, I also ran into this issue.

With two files as a test from Smithsonian 3D - a largish 49.2 Mbyte STL file, and a 12.9 Mbyte STL file. Both resulted in "Internal error: Server failed to store temporary file."

matthiasmullie closed this task as Resolved.May 15 2017, 12:34 PM

I believe this is now fixed.
Given the timing, I guess it may have been related to the DC switchover, but I'm not sure.
Either way, it's working for me, both with (the same) STL & JPG file it used to fail with before.
Please reopen this thread should you still experience this issue.

• Elitre awarded a token.May 15 2017, 12:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL