Implement http-less file-copy functionality
Open, Stalled, NormalPublic

Description

To avoid unneeded traffic in areas of the cluster we want to be able to copy the files directly without http.

Addshore created this task.Mar 21 2017, 6:02 PM
Addshore claimed this task.
Restricted Application added a project: User-Addshore. · View Herald TranscriptApr 11 2017, 3:20 PM
Addshore moved this task from Backlog to In Progress on the User-Addshore board.Apr 11 2017, 3:20 PM
Addshore moved this task from Todo to Sprint ready on the WMDE-QWERTY-Team-Board board.
Addshore added a comment.EditedMay 30 2017, 2:04 PM

So, my notes from the hackathon are:

ForeignDbViaLBRepo
Getbackendpath somewhere?
Filebackend::copy

Not 100% sure about the impact this will have on the current method of using the built in mediawiki importing stuff :/

Change 364253 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/FileImporter@master] WIP DNM moving without http

https://gerrit.wikimedia.org/r/364253

Addshore changed the task status from Open to Stalled.
Tobi_WMDE_SW renamed this task from Implement Swift based file-copy functionality to Implement http-less file-copy functionality.Aug 22 2017, 3:00 PM
Tobi_WMDE_SW updated the task description. (Show Details)
Tobi_WMDE_SW moved this task from Sprint Backlog to Doing on the WMDE-QWERTY-Sprint-2017-08-22 board.

The code for this is basically done, but testing this functionality locally is hardly possible. So we decide to wait for the security review (T160982) and test it once it is deployed to beta.

I didn't fully read the code, though I'm curious what happens to the files on the FileBackend side of things and specifically to swift in production/beta. Thanks!

@fgiunchedi so the bit of code to look at is the commit method of https://gerrit.wikimedia.org/r/#/c/364253/11/src/Operations/FileRevisionFromForeignDbFile.php

This calls LocalFile::upload() with the backend storage path of the file to copy
This ends up calling the following AFAIK. LocalFile::publish -> LocalFile::publishTo -> FileRepo::quickImport -> FileRepo::quickImportBatch (which is where the actual backend operations are created.

From the testing that I have done this should all work, although I never managed to get any sort of swift setup similar to that in production locally or on labs to test this out.

There's nothing to do until we have the extension on beta because we cannot test it on local environments.