Page MenuHomePhabricator

Can asynchronous side-loading be activated on Commons?
Open, NormalPublic

Description

Especially with video we need a feature to regularly upload bigger files.

Thanks to chunked uploads and CopyUploads we can upload files up to 1 GB if the user is able to jump through some hoops (having Image Reviewer or GW Toolset permissions, owning a server or having the ability to operate a bot).

But even for these experienced users there is no possibility to upload files bigger than 1 GB, like these:

A fix for this situation - without having to file bugs every time and pray for a developer that has time to deal with the uploads - could be to allow asynchronous Copy Uploads. For some reason they are disabled and the feature is described as "experimental" but wouldn't it be worth to activate and test it on Wikimedia Commons?

https://www.mediawiki.org/wiki/Manual:$wgAllowAsyncCopyUploads

Example of an attempted Copy Upload with phpapibot (debug output):

  • dumping poststring:

array(6) {

["url"]=>
string(124) "http://upload-80686.wikimedia.ch/2014-10-23%20Zeitzeugen%20-%2025%20Jahre%20Berliner%20Mauerfall%20-%20Armin%20Schuster.webm"
["filename"]=>
string(73) "2014-10-23 Zeitzeugen - 25 Jahre Berliner Mauerfall - Armin Schuster.webm"
["text"]=>
string(11) "upload-text"
["asyncdownload"]=>
string(1) "1"
["ignorewarnings"]=>
string(1) "1"
["token"]=>
string(34) "eb6b46544aa32e2c0d2ae101d533d022+\"

}


  • DEBUG: result string

a:2:{s:8:"servedby";s:6:"mw1201";s:5:"error";a:3:{s:4:"code";s:23:"asynccopyuploaddisabled";s:4:"info";s:34:"Asynchronous copy uploads disabled";s:1:"*";s:56:"See http://commons.wikimedia.org/w/api.php for API usage";}}

  • DEBUG: unserialized object:

array(3) {

["servedby"]=>
string(6) "mw1201"
["error"]=>
array(3) {
  ["code"]=>
  string(23) "asynccopyuploaddisabled"
  ["info"]=>
  string(34) "Asynchronous copy uploads disabled"
  ["*"]=>
  string(56) "See http://commons.wikimedia.org/w/api.php for API usage"
}
["token"]=>
string(34) "eb6b46544aa32e2c0d2ae101d533d022+\"

}


Version: unspecified
Severity: enhancement

Details

Reference
bz72531

Event Timeline

bzimport raised the priority of this task from to Normal.Nov 22 2014, 3:51 AM
bzimport set Reference to bz72531.
bzimport added a subscriber: Unknown Object (MLST).
80686 created this task.Oct 26 2014, 3:13 PM
80686 set Security to None.
80686 added a comment.May 15 2015, 8:32 AM

may I ask for input / feedback on that task please?

Dereckson added a subscriber: csteipp.EditedMay 15 2015, 9:33 AM

Hi,

Yes, it could be interesting to test this feature, especially now you've confirmed it works and is valuable on upload-80686.wikimedia.ch.

@csteipp any objection we deploy this on Labs at Commons beta to test it?

Well for starters, https://www.mediawiki.org/wiki/Special:Code/MediaWiki/81615 stands in the way of this.

The feature was experimental when it was written. Since then its been hard disabled in code, and nobody has tested it, so its old and experimental. I'm not confident the feature is currently in good enough shape to be enabled on a production site even after the obvious issue with the if statement is fixed.

In order to move forward, I believe a developer would have to review the code, rectify any deficiencies, and ensure it is up to modern mediawiki standards.

I'm also somewhat doubtful we would want to have the side loading limit be any higher than the chunked upload limit. Thus I don't think you would get around the 1 gb limit this way unless the limit was raised for other upload methods. I would also note, that even if this feature was enabled right now, all side loading uploads (synchronous or otherwise) go through url-downloader.wikimedia.org, which places a 1gb limit on file size.

Hi,
Yes, it could be interesting to test this feature, especially now you've confirmed it works and is valuable on upload-80686.wikimedia.ch.

He didn't confirm it works. The transcript posted is of making a request to commons, and then commons responding by saying the feature is disabled.

(Oh, sorry, I've read too fast the transcript and imagined it were a comparison between Commons and a 3rd party test wiki.)

80686 added a comment.EditedMay 15 2015, 10:23 AM

Thanks for your comments.

It mainly affects video uploads and for that we have the video editing server in the meantime, which can move stuff to Commons - see https://phabricator.wikimedia.org/T99216

So we should evaluate first if the effort to get that code reviewed (and maybe even rewritten) is worth it.

We could consider though that other people are using that feature in their own wikis, so getting that code cleaned up is a good thing in general.

Thanks for your comments.
It mainly affects video uploads and for that we have the video editing server in the meantime, which can move stuff to Commons - see https://phabricator.wikimedia.org/T99216
So we should evaluate first if the effort to get that code reviewed (and maybe even rewritten) is worth it.
We could consider though that other people are using that feature in their own wikis, so getting that code cleaned up is a good thing in general.

Well worth it, and have someone willing to do it. I think there's a lot more video related tasks that are worth it then there is available labour. Actually, from the mediawiki prespective, it seems like available labour for new video features (as opposed to bug fixing things already deployed), is close to 0 (I would love to be proven wrong on that).

revi added a subscriber: revi.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptDec 12 2015, 8:37 AM
Restricted Application added a subscriber: JEumerus. · View Herald TranscriptJan 22 2016, 8:47 PM
matmarex changed the status of subtask T119336: Fix async upload by url from Resolved to Declined.Jan 22 2016, 8:57 PM
Yann added a subscriber: Yann.Mar 19 2016, 10:46 PM
scfc added a subscriber: scfc.Feb 23 2017, 2:40 PM