In the metadata detection field I uploaded the xml file. I've tried two xml files: one that previously worked and a new one. After hitting 'submit' both didn't go through and gave the same error message (as mentioned above).

Error is: <p>Please contact a developer. This issue must be addressed before you can continue. Please add the following text to your report: </p><p>GWToolset\Helpers\GWTFileBackend::saveFile: An unknown error occurred in storage backend "local-swift-codfw". </p>

So something to do with recent data center testing?

I'm doing my job today as a Wikimedian In Residence. I don't often have access to the server which generates the XML, and I've come today to the office from where I have access specially to do some uploads, although I'll have access tomorrow, so for me this is an urgent problem.

I have also tried two different XML files, one new and a previous one that has worked before. The error I get each time on uploading the XML is "[0453e274] 2016-03-10 15:00:35: Fatal exception of type "MWException"" Thanks in advance for fixing this.

From logstash: 70ed1b6b on mw1214 at 2016-03-09T15:15:49.000Z

{"file":"/srv/mediawiki/php-1.27.0-wmf.15/extensions/GWToolset/includes/Handlers/UploadHandler.php","line":643,"function":"saveFile","class":"GWToolset\\Helpers\\GWTFileBackend","type":"->","args":["Php\\File"]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/extensions/GWToolset/includes/Handlers/Forms/MetadataDetectHandler.php","line":205,"function":"saveMetadataToFileBackend","class":"GWToolset\\Handlers\\UploadHandler","type":"->","args":[]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/extensions/GWToolset/includes/Handlers/Forms/FormHandler.php","line":177,"function":"processRequest","class":"GWToolset\\Handlers\\Forms\\MetadataDetectHandler","type":"->","args":[]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/extensions/GWToolset/includes/Specials/SpecialGWToolset.php","line":123,"function":"execute","class":"GWToolset\\Handlers\\Forms\\FormHandler","type":"->","args":[]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/extensions/GWToolset/includes/Specials/SpecialGWToolset.php","line":76,"function":"processRequest","class":"GWToolset\\SpecialGWToolset","type":"->","args":[]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/includes/specialpage/SpecialPage.php","line":407,"function":"execute","class":"GWToolset\\SpecialGWToolset","type":"->","args":["NULL"]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/includes/specialpage/SpecialPageFactory.php","line":565,"function":"run","class":"SpecialPage","type":"->","args":["NULL"]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/includes/MediaWiki.php","line":282,"function":"executePath","class":"SpecialPageFactory","type":"::","args":["Title","RequestContext"]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/includes/MediaWiki.php","line":738,"function":"performRequest","class":"MediaWiki","type":"->","args":[]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/includes/MediaWiki.php","line":519,"function":"main","class":"MediaWiki","type":"->","args":[]},
{"file":"/srv/mediawiki/php-1.27.0-wmf.15/index.php","line":43,"function":"run","class":"MediaWiki","type":"->","args":[]},
{"file":"/srv/mediawiki/w/index.php","line":3,"function":"include","args":["string"]}

Which seems to actually point to the following error:

HTTP 404 (Not Found) in 'SwiftFileBackend::doStoreInternal' (given '{"async":false,"op":"store","src":"/tmp/74fkIy","dst":"mwstore://local-swift-codfw/gwtoolset-metadata/85jesse/3/c/4/3c4824f0616b71641e9fa44b15805f7a.xml","overwrite":true}')

I wonder if this is something silly, like someone forgot to create a container for gwtoolset-metadata in codfw, and enabling replication in 5fa77bd751998 broke it.

(ping @aaron )

A PUT to a non-existing container gives 404s, though code should call prepare() before doOperations() to avoid this problem (as other callers do). It's like making a file without making the directory first.

GWtoolset does seem to make such a call prior to doQuickStore()

return $this->FileBackend->prepare(
        array(
                'dir' => $this->getMWStoreFileDirectory(),
                'noAccess' => true,
                'noListing' => true
        )
);

Mentioned in SAL [2016-03-10T18:46:23Z] <godog> sync wikipedia-commons-gwtoolset-metadata with swiftrepl eqiad -> codfw T129359

indeed that container doesn't match the usual list of containers that we synced, I'm syncing it now with swiftrepl

Hmm, so local-swift-codfw has async writes. thus ->prepare() should be delayed until the end of the request. But the quickStore() is a store op, so that would not be deferred. Thus quickStore will always happen first. So quickStore fails since prepare() hasn't run yet, which causes gwtoolset to throw an exception. But that would only explain the issue happening once. Even with the exception, the deferred update should still run (I think)

{{done}}, could you try again @Bawolff @MartinPoulter @85jesse