Page MenuHomePhabricator

Upload by URL should use the job queue, possibly chunked with range requests
Open, Needs TriagePublic

Description

I would probably recommend retooling the upload via URL to run via the job queue, like long-running video transcodes do. This would allow for downloads to take longer if needed, without the possibility of dosing the app servers. This is a bigger task though...

But more to my point: what should change is the fact that upload-by-url can take up to 200 seconds to complete a single HTTP request.
HTTP requests should terminate reasonably quickly and the work should be done in the background via the jobqueue, and the client should have a way of polling it.

I also discussed this with @tstarling recently, if a single long-running job doesn't work, we could split it into multiple jobs using range requests similar to how chunked uploads work.