Change 422183 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] [WIP] Describe command and data flow in README

https://gerrit.wikimedia.org/r/422183

@Joe @fgiunchedi Would probably be a good idea if one/both of you took a look at this, just to be sure that the choices being made around where to put temp files, limits on size and quantity, etc are all reasonable.

@Gilles We decided this didn't really need a perf review, but if you have a few minutes free it couldn't hurt to look things over.

Looking through the tickets and the Readme of the extension, I think the default configurations are somewhat sane, but I'd reduce the total size of the images that can be fetched probably.

I don't think there is time to change the way the extension works before deployment, but I think we would have better performance and a more sound architecture if we used Swift's COPY function to copy images to their final destination in commons; @Lea_WMDE do you think that's a direction we can explore for a future enhancement?

A Swift COPY is possible, but would require exposing sharding information of all wikis to a given wiki's config. Right now each wiki only gets information about its own sharding. It also requires significant refactoring or trickery in the SwiftFileBackend family of classes, which is currently architected around dealing with one wiki-specific "FileBackend" at a time.

In T190716#4224000, @Joe wrote:

Looking through the tickets and the Readme of the extension, I think the default configurations are somewhat sane, but I'd reduce the total size of the images that can be fetched probably.

Would you want the limit reduced now already? We plan to deploy to de-wiki and one RTL-wiki as a first step, and will log the sizes people will actually try to move. We would have waited until we have some data to reevaluate the limits, but if you feel we should start with another size limit, let me know which one it should be :)

I don't think there is time to change the way the extension works before deployment, but I think we would have better performance and a more sound architecture if we used Swift's COPY function to copy images to their final destination in commons; @Lea_WMDE do you think that's a direction we can explore for a future enhancement?

Generally we should invest time to get better performance and a more sound architecture. However, I propose to reevaluate that topic once we have more data on how much load is actually created. @Addshore looked into using Swift's COPY function quite a bit already, but without success. So before we look into it again, we should have some more discussions with you, him, @Gilles and other knowledgeable people to make sure this can actually work. I suggest we do that once we feel like it is worth to put in the effort in comparison to the load that is being created.

Shortly:

• It's ok to reevaluate the limits after we deploy, 250 MB is not a huge amount of data and will save us from some of the most absurd situations we have, like huge tiff or djvu collections being moved and killing an appserver and/or swift in the process.
• I'm ok with the approach taken for now, and I think it's fair to want to know how much this feature will actually be used before pouring more resources into it. I just intuitively assumed it would be not such a daunting task, but I get why it's not straightforward to do now that I looked at the code for SwiftFileBackend.

Change 436356 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/FileImporter@master] [WIP] Add documentation about throttling and alternative backends

https://gerrit.wikimedia.org/r/436356

Change 436356 merged by jenkins-bot:
[mediawiki/extensions/FileImporter@master] Add documentation about throttling and alternative backends

https://gerrit.wikimedia.org/r/436356