To avoid unneeded traffic in areas of the cluster we want to be able to copy the files directly without http.
Description
Details
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
WIP Moving without http | mediawiki/extensions/FileImporter | master | +569 -75 |
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | None | T140462 Correctly move files from Wikipedia to Commons (2013) | |||
Resolved | thiemowmde | T146666 Build a commons extension that moves files to Commons | |||
Duplicate | None | T156798 Implement the file-copy functionality | |||
Declined | Addshore | T161012 Implement http-less file-copy functionality | |||
Resolved | Addshore | T185671 Test file importer on beta |
Event Timeline
So, my notes from the hackathon are:
ForeignDbViaLBRepo
Getbackendpath somewhere?
Filebackend::copy
Not 100% sure about the impact this will have on the current method of using the built in mediawiki importing stuff :/
Change 364253 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/FileImporter@master] WIP DNM moving without http
The code for this is basically done, but testing this functionality locally is hardly possible. So we decide to wait for the security review (T160982) and test it once it is deployed to beta.
I didn't fully read the code, though I'm curious what happens to the files on the FileBackend side of things and specifically to swift in production/beta. Thanks!
@fgiunchedi so the bit of code to look at is the commit method of https://gerrit.wikimedia.org/r/#/c/364253/11/src/Operations/FileRevisionFromForeignDbFile.php
This calls LocalFile::upload() with the backend storage path of the file to copy
This ends up calling the following AFAIK. LocalFile::publish -> LocalFile::publishTo -> FileRepo::quickImport -> FileRepo::quickImportBatch (which is where the actual backend operations are created.
From the testing that I have done this should all work, although I never managed to get any sort of swift setup similar to that in production locally or on labs to test this out.
There's nothing to do until we have the extension on beta because we cannot test it on local environments.