From andrew when first spotted:

And the debug logs shows:

This behavior would make sense if there were left over files from a previous failed import but I explicitly used ./down (docker system) and cleared the mediawiki/images folder.

Weird enough when I try to upload the file again it fails with a different file name conflict:

So, it looks like the issue is coming from FileOp line 390 in core,

$status->fatal( 'backend-fail-alreadyexists', $this->params['dst'] );

This happens on a totally fresh install / totally clean file backend, so need to do some more investigation / stepping through this!

Looked a bit into it and it always seems to fail when trying to import
https://upload.wikimedia.org/wikipedia/commons/archive/8/8c/20151121185130%21Labs_Tools_topology.png
"Update for current state of tool labs"

Okay, so I found another image which seems to react in the same way:
https://commons.wikimedia.org/wiki/File:St._Michael_ob_Rauchen%C3%B6dt_Fl%C3%BCgelaltar_01.jpg.

It looks like this is still happening, tested on https://test.wikimedia.beta.wmflabs.org/wiki/Special:ImportFile

[WoMEcwpEE4AAADivxrAAAAAQ] /wiki/Special:ImportFile RuntimeException from line 109 of /srv/mediawiki/php-master/extensions/FileImporter/src/Services/Importer.php: Failed to commit operations.

Backtrace:

#0 /srv/mediawiki/php-master/extensions/FileImporter/src/SpecialImportFile.php(238): FileImporter\Services\Importer->import(User, FileImporter\Data\ImportPlan)
#1 /srv/mediawiki/php-master/extensions/FileImporter/src/SpecialImportFile.php(144): FileImporter\SpecialImportFile->doImport(FileImporter\Data\ImportPlan)
#2 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(522): FileImporter\SpecialImportFile->execute(NULL)
#3 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(579): SpecialPage->run(NULL)
#4 /srv/mediawiki/php-master/includes/MediaWiki.php(288): SpecialPageFactory::executePath(Title, RequestContext)
#5 /srv/mediawiki/php-master/includes/MediaWiki.php(861): MediaWiki->performRequest()
#6 /srv/mediawiki/php-master/includes/MediaWiki.php(524): MediaWiki->main()
#7 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#8 /srv/mediawiki/w/index.php(3): include(string)
#9 {main}

In T176871#3773983, @Andrew-WMDE wrote:

Okay, so I found another image which seems to react in the same way:
https://commons.wikimedia.org/wiki/File:St._Michael_ob_Rauchen%C3%B6dt_Fl%C3%BCgelaltar_01.jpg.

This example now works.

One thing I ran into the last time looking into this is

$archiveName = wfTimestamp( TS_MW ) . '!' . $this->getName();

in LocalFile::publishTo() I am not sure is this could cause problems when importing the archived files very quickly.

https://gerrit.wikimedia.org/g/mediawiki/core/+/master/includes/filerepo/file/LocalFile.php#1861

I think this is it. When I look at an example file like https://commons.wikimedia.org/wiki/File:Test.svg, the versions are named like this:

• https://upload.wikimedia.org/wikipedia/commons/archive/b/bd/20180416165039!Test.svg
• https://upload.wikimedia.org/wikipedia/commons/archive/b/bd/20180416164953!Test.svg

That's down to a second, without a random seed to avoid conflicts when two actions happen the same second. The LocalFile class contains two places where a filename like this is build: One actively tries to increment the timestamp to avoid conflicts. One doesn't, but simply uses the current timestamp:

function publishTo( … ) {
	…
	$archiveName = wfTimestamp( TS_MW ) . '!' . $this->getName();
	$archiveRel = 'archive/' . $this->getHashPath() . $archiveName;

There is also one line of code in ImportableUploadRevisionImporter::import that builds such an archive file name.

Note the timestamps in the filenames are not the timestamps when the file revision was originally uploaded! An example can be seen at https://commons.wikimedia.org/wiki/File:Joan_of_Arc_miniature_graded.jpg. I guess the archive names are sometimes updated when an archived file revision is restored, for example.

I was able to prove my idea locally: With the file https://commons.wikimedia.org/wiki/File:Joan_of_Arc_miniature_graded.jpg I can reproduce the exception. The moment I added a sleep( 1 ) in FileRevisionFromRemoteUrl::commit the import succeeded.

Next issue: The history of file versions is totally messed up in my tests. All file versions point to the same file name in the archive. Can anybody confirm this?

Change 426959 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/core@master] Fix bad archive file names in ImportableUploadRevisionImporter

https://gerrit.wikimedia.org/r/426959

Change 426960 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] [DNM] Experiment with sleep( 1 ) to avoid archive clash

https://gerrit.wikimedia.org/r/426960

In T176871#4134331, @gerritbot wrote:

Change 426959 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/core@master] Fix bad archive file names in ImportableUploadRevisionImporter

https://gerrit.wikimedia.org/r/426959

That patch works pretty damn well for me here :-)! The file can be imported with no errors and looks fine .... so maybe that's it?

Change 426959 merged by jenkins-bot:
[mediawiki/core@master] Fix bad archive file names in ImportableUploadRevisionImporter

https://gerrit.wikimedia.org/r/426959

Could be tested on http://mwfileimport.wmflabs.org/wiki/index.php/Special:ImportFile

Change 427438 had a related patch set uploaded (by Krinkle; owner: Thiemo Kreuz (WMDE)):
[mediawiki/core@REL1_31] Fix bad archive file names in ImportableUploadRevisionImporter

https://gerrit.wikimedia.org/r/427438

Change 427438 merged by jenkins-bot:
[mediawiki/core@REL1_31] Fix bad archive file names in ImportableUploadRevisionImporter

https://gerrit.wikimedia.org/r/427438

Is this resolved?

@demon It is done but still in an ongoing sprint. So we will close it on Wednesday :)

Change 426960 abandoned by Thiemo Kreuz (WMDE):
[DNM] Experiment with sleep( 1 ) to avoid archive clash

https://gerrit.wikimedia.org/r/426960