Page MenuHomePhabricator

Implement http-less file-copy functionality
Closed, DeclinedPublic

Description

To avoid unneeded traffic in areas of the cluster we want to be able to copy the files directly without http.

Event Timeline

Addshore moved this task from Proposed to Todo on the WMDE-TechWish board.

So, my notes from the hackathon are:

ForeignDbViaLBRepo
Getbackendpath somewhere?
Filebackend::copy

Not 100% sure about the impact this will have on the current method of using the built in mediawiki importing stuff :/

Change 364253 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/FileImporter@master] WIP DNM moving without http

https://gerrit.wikimedia.org/r/364253

Tobi_WMDE_SW renamed this task from Implement Swift based file-copy functionality to Implement http-less file-copy functionality.Aug 22 2017, 3:00 PM
Tobi_WMDE_SW updated the task description. (Show Details)
Tobi_WMDE_SW moved this task from Sprint Backlog to Doing on the WMDE-QWERTY-Sprint-2017-08-22 board.

The code for this is basically done, but testing this functionality locally is hardly possible. So we decide to wait for the security review (T160982) and test it once it is deployed to beta.

I didn't fully read the code, though I'm curious what happens to the files on the FileBackend side of things and specifically to swift in production/beta. Thanks!

@fgiunchedi so the bit of code to look at is the commit method of https://gerrit.wikimedia.org/r/#/c/364253/11/src/Operations/FileRevisionFromForeignDbFile.php

This calls LocalFile::upload() with the backend storage path of the file to copy
This ends up calling the following AFAIK. LocalFile::publish -> LocalFile::publishTo -> FileRepo::quickImport -> FileRepo::quickImportBatch (which is where the actual backend operations are created.

From the testing that I have done this should all work, although I never managed to get any sort of swift setup similar to that in production locally or on labs to test this out.

There's nothing to do until we have the extension on beta because we cannot test it on local environments.

Change 364253 abandoned by Addshore:
WIP Moving without http

https://gerrit.wikimedia.org/r/364253