Page MenuHomePhabricator
Create Task
Maniphest T295007

Upload by URL should use the job queue, possibly chunked with range requests
Closed, ResolvedPublic
Actions

Description

In T118887#2233280, @brion wrote:

I would probably recommend retooling the upload via URL to run via the job queue, like long-running video transcodes do. This would allow for downloads to take longer if needed, without the possibility of dosing the app servers. This is a bigger task though...

In T294800#7480583, @Joe wrote:

But more to my point: what should change is the fact that upload-by-url can take up to 200 seconds to complete a single HTTP request.
HTTP requests should terminate reasonably quickly and the work should be done in the background via the jobqueue, and the client should have a way of polling it.

I also discussed this with @tstarling recently, if a single long-running job doesn't work, we could split it into multiple jobs using range requests similar to how chunked uploads work.

Details

SubjectRepoBranchLines +/-
Enable async jobqueue-powered URL uploads on commonsoperations/mediawiki-configmaster+1 -0
Enable async upload-by-URL via jobqueue on testwikioperations/mediawiki-configmaster+7 -1
Enable async upload-by-URL on testwikioperations/mediawiki-configmaster+5 -0
Switch Special:Upload to use async upload-by-urlmediawiki/coremaster+450 -27
Add job for upload from UploadFromUrlmediawiki/coremaster+102 -4
Allow async upload by url via the Apimediawiki/coremaster+96 -27
Customize query in gerrit

Related Objects
Search...

Event Timeline

Legoktm created this task.Nov 4 2021, 7:45 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 4 2021, 7:45 AM
Legoktm added a parent task: T118887: Upload by URL doesn't work well for large files: HTTP request timed out..Nov 4 2021, 7:50 AM
Legoktm mentioned this in T280926: Various errors when trying to upload large files (Could not acquire lock, Service Temporarily Unavailable, 503 Backend fetch failed, 502 Next Hop Connection Failed).Nov 4 2021, 8:11 AM
Legoktm mentioned this in T292954: [epic] large file uploads to commons.Nov 4 2021, 8:17 AM
Yann subscribed.Nov 4 2021, 4:04 PM
Legoktm mentioned this in T292322: Support large files in Shellbox.Feb 1 2022, 6:14 PM
• dpifke subscribed.Feb 10 2022, 5:05 PM
Joe added a project: MediaWiki CodeJam Dec 2023.Dec 6 2023, 2:43 PM
Ladsgroup subscribed.Dec 6 2023, 2:50 PM
Joe triaged this task as High priority.Dec 6 2023, 3:50 PM

We intend to try to take a stab at this during next week's MediaWiki CodeJam.

It will if anything be a chance for me to refresh my mediawiki internals knowledge and for others to mock my ability to interact with a web frontend :P

gerritbot added a comment.Dec 13 2023, 8:08 AM

Change 982757 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[mediawiki/core@master] Add job for upload from UploadFromUrl.

https://gerrit.wikimedia.org/r/982757

gerritbot added a project: Patch-For-Review.Dec 13 2023, 8:08 AM
gerritbot added a comment.Dec 14 2023, 2:22 PM

Change 983196 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[mediawiki/core@master] Allow async upload by url via the Api

https://gerrit.wikimedia.org/r/983196

Yann added a comment.Feb 12 2024, 4:44 PM

What's the status of this task?

gerritbot added a comment.Mar 23 2024, 10:29 AM

Change #1007344 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[mediawiki/core@master] Switch Special:Upload to use async upload-by-url

https://gerrit.wikimedia.org/r/1007344

Joe added a comment.Mar 23 2024, 10:40 AM
In T295007#9534555, @Yann wrote:

What's the status of this task?

Hi, sorry for not replying earlier but this work is kind-of a side hustle for me - as you might have noticed, this stuff is far from my area of expertise :)

I have completed the basic work to make this work both in Special:Upload and in the API. While I think we can merge the patches to add asynchronous behaviour to the API as soon as I find a reviewer, the Special:Upload stuff will need me to find someone to help me polishing the UI part of it.

Doing frontend web development is really both not my job nor my area of expertise.

With the API change, you should be able to upload by url large files (up to 4 GB IIRC) without incurring in timeouts; and that will also allow us to move file processing to Shellbox, making our infrastructure more secure.

Here is an example of a script that works with the async api: P58902

Would this be enough to unblock people who want to upload large files, while I try to polish the Special:Upload patch?

Jeff_G subscribed.Mar 23 2024, 4:05 PM
In T295007#9655859, @Joe wrote:

With the API change, you should be able to upload by url large files (up to 4 GB IIRC) without incurring in timeouts; and that will also allow us to move file processing to Shellbox, making our infrastructure more secure.

@Joe: How about 5 GiB per https://gerrit.wikimedia.org/r/1002813 ? See https://phabricator.wikimedia.org/T191804#9363066 for a discussion
about the capacity implications of this change.

Joe added a comment.Mar 24 2024, 7:39 AM
In T295007#9655915, @Jeff_G wrote:
In T295007#9655859, @Joe wrote:

With the API change, you should be able to upload by url large files (up to 4 GB IIRC) without incurring in timeouts; and that will also allow us to move file processing to Shellbox, making our infrastructure more secure.

@Joe: How about 5 GiB per https://gerrit.wikimedia.org/r/1002813 ? See https://phabricator.wikimedia.org/T191804#9363066 for a discussion
about the capacity implications of this change.

Once we've gone completely async and we've confirmed it works, it might be a good idea to increase the allowed size for files that are uploaded by URL.

As-is, the synchronous process doesn't allow you to upload files larger than 2-3GB because it often times out before a larger file is processed. So even getting to the current official limit of 4 GB and make it work reliably seems like a good first step.

gerritbot added a comment.Mar 25 2024, 11:37 AM

Change #982757 merged by jenkins-bot:

[mediawiki/core@master] Add job for upload from UploadFromUrl

https://gerrit.wikimedia.org/r/982757

gerritbot added a comment.Mar 25 2024, 11:41 AM

Change #983196 merged by jenkins-bot:

[mediawiki/core@master] Allow async upload by url via the Api

https://gerrit.wikimedia.org/r/983196

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.24; 2024-03-26).Mar 25 2024, 12:00 PM
gerritbot added a comment.Mar 28 2024, 3:22 PM

Change #1007344 merged by jenkins-bot:

[mediawiki/core@master] Switch Special:Upload to use async upload-by-url

https://gerrit.wikimedia.org/r/1007344

Ladsgroup added a comment.Mar 29 2024, 10:42 PM

Let's enable this in beta cluster. If anyone is feeling like it.

Clement_Goubert mentioned this in T362323: Move 100% of external traffic to Kubernetes (excluding Votewiki).Apr 11 2024, 12:46 PM
gerritbot added a comment.Apr 26 2024, 3:41 PM

Change #1024731 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/mediawiki-config@master] Enable async upload-by-URL on testwiki

https://gerrit.wikimedia.org/r/1024731

hnowlan subscribed.Apr 26 2024, 4:01 PM
gerritbot added a comment.Apr 29 2024, 12:43 PM

Change #1024731 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable async upload-by-URL on testwiki

https://gerrit.wikimedia.org/r/1024731

Maintenance_bot removed a project: Patch-For-Review.Apr 29 2024, 1:31 PM
gerritbot added a comment.Apr 30 2024, 2:07 PM

Change #1025790 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/mediawiki-config@master] Enable async upload-by-URL via jobqueue on commons

https://gerrit.wikimedia.org/r/1025790

gerritbot added a project: Patch-For-Review.Apr 30 2024, 2:07 PM
gerritbot added a comment.Mon, May 13, 1:47 PM

Change #1025790 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable async upload-by-URL via jobqueue on testwiki

https://gerrit.wikimedia.org/r/1025790

Stashbot added a comment.Mon, May 13, 1:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-13T13:47:20Z] <logmsgbot> lucaswerkmeister-wmde@deploy1002 Started scap: Backport for [[gerrit:1025790|Enable async upload-by-URL via jobqueue on testwiki (T295007)]]

Stashbot added a comment.Mon, May 13, 1:49 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-13T13:49:46Z] <logmsgbot> lucaswerkmeister-wmde@deploy1002 hnowlan and lucaswerkmeister-wmde: Backport for [[gerrit:1025790|Enable async upload-by-URL via jobqueue on testwiki (T295007)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Mon, May 13, 2:12 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-13T14:12:30Z] <logmsgbot> lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for [[gerrit:1025790|Enable async upload-by-URL via jobqueue on testwiki (T295007)]] (duration: 25m 09s)

Maintenance_bot removed a project: Patch-For-Review.Mon, May 13, 2:31 PM
gerritbot added a comment.Mon, May 13, 5:02 PM

Change #1031028 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/mediawiki-config@master] Enable async jobqueue-powered URL uploads on commons

https://gerrit.wikimedia.org/r/1031028

gerritbot added a project: Patch-For-Review.Mon, May 13, 5:02 PM
hnowlan added a comment.Wed, May 15, 10:40 AM

Testing of this feature on testwiki has been successful so far, we are hoping to enable it on commons tomorrow in the UTC backport window. If there are any pending large URL uploads waiting to be rolled out it would be very handy to test using these

gerritbot added a comment.Thu, May 16, 1:13 PM

Change #1031028 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable async jobqueue-powered URL uploads on commons

https://gerrit.wikimedia.org/r/1031028

Stashbot added a comment.Thu, May 16, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T13:14:11Z] <jsn@deploy1002> Started scap: Backport for [[gerrit:1031028|Enable async jobqueue-powered URL uploads on commons (T295007)]]

Stashbot added a comment.Thu, May 16, 1:16 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T13:16:52Z] <jsn@deploy1002> jsn and hnowlan: Backport for [[gerrit:1031028|Enable async jobqueue-powered URL uploads on commons (T295007)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Maintenance_bot removed a project: Patch-For-Review.Thu, May 16, 1:31 PM
Stashbot added a comment.Thu, May 16, 1:32 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T13:32:29Z] <jsn@deploy1002> Finished scap: Backport for [[gerrit:1031028|Enable async jobqueue-powered URL uploads on commons (T295007)]] (duration: 18m 18s)

hnowlan added a comment.Thu, May 16, 1:47 PM

This feature is now live on commons and appears to be functioning correctly. Please update ASAP if you notice anything amiss. I'd be particularly interested to know about success or failure of large file uploads.

hnowlan closed this task as Resolved.Mon, May 20, 3:38 PM
hnowlan claimed this task.

I am going to resolve this ticket for now - please reopen if you notice any issues

• Lucas_Werkmeister_WMDE subscribed.Fri, May 24, 1:42 PM

I’m not aware of any issues, but if this is working (great!), then we should probably enable it by default in MediaWiki, and everywhere in production? Right now AFAICT $wgEnableAsyncUploadsByURL still defaults to false in core and is only enabled in commonswiki, testwiki and the Beta Cluster in wmf-config.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL