Page MenuHomePhabricator

Set $wgEnableAsyncUploads = true on all wikis
Closed, ResolvedPublic

Description

Recently on T364288 there are complaints that large uploads were not working on english wikipedia. The issue turns out to be that $wgEnableAsyncUploads is false on that wiki, so uploads must complete within the time limit of a POST request.

The current config is:

// Wikis which allow async uploads via the job queue if uploads are enabled
'wgEnableAsyncUploads' => [
	'default' => false,
	'commonswiki' => true,
	'testcommonswiki' => true,
	'testwiki' => true,
	'test2wiki' => true,
],

// Raise stash time limit on Commons for UploadWizard
'wmgUploadStashMaxAge' => [
	'default' => 6 * 60 * 60, // 6 hours
	'commonswiki' => 48 * 60 * 60, // 48 hours
	'testcommonswiki' => 48 * 60 * 60, // 48 hours
],

I don't understand why we are limiting this to commons. As far as I know, we have the job queue set up for all wikis. modules/profile/manifests/mediawiki/maintenance/cleanup_upload_stash.pp seems to run the maintenance script on all wikis

Event Timeline

I have no idea if anyone is still around who knows the history of why this is the way it is. @Sannita does the current upload wizard team have any thoughts or objections to changing this?

Hi, sorry for the late reply, but I was offline this weekend. I'll ask around, I think site reliability is also involved in this, and I would like to have their opinion too.

I'm not entirely sure why this is defaulted to false, but it has been the case since at least 2013. Personally I don't know of any good reason not to enable it on other wikis - I can see an increased impact on the jobqueue but we can scale that up if needs be if this is an overall quality of life improvement for the wikis. I would worry that there is some history I am unaware of though.

I suspect it wont even be that much more increase on job queue since upload wizard is disabled on other wikis. This would entirely be about third party tools which is probably just a few people.

@aaron disabled it on 2013-03-14 and re-enabled it for Commons only on 2013-03-29. But there was no UI for it on wikis other than Commons, and that remains the case.

Note that in January of that year there was 44693 and 45282, disabling this feature in other parts of the stack, in response to T46080.

What would be the usecase for upload from other sites to non-commons. Assuming they are all fair use images (otherwise they should go to commons), maybe we shouldn't allow it? I know there are some discrepancies between copyright laws and some wikis allow some images that wouldn't be allowed in Commons but that's still quite a marginal usecase (unless I'm missing something super obvious). e.g. for T364288: Server-side upload request for hinnk why it's not being uploaded to commons?

Files that are public domain in the U.S. but not in the country of origin are allowed on English Wikipedia but not Wikimedia Commons.

This is the case for a lot of silent feature films. The Adventures of Prince Achmed was the first I tried to upload, but there are several more I have queued up, enough that requesting server-side uploads is not a feasible strategy because the total size would be larger than my Google storage space.

What would be the usecase for upload from other sites to non-commons. Assuming they are all fair use images (otherwise they should go to commons), maybe we shouldn't allow it? I know there are some discrepancies between copyright laws and some wikis allow some images that wouldn't be allowed in Commons but that's still quite a marginal usecase (unless I'm missing something super obvious). e.g. for T364288: Server-side upload request for hinnk why it's not being uploaded to commons?

As long as there is some marginal use case i think we should do it. Individual communities can use abuse filter to restrict things as need be. Then any restrictions would be in the hands of the community, who are better positioned to deal with the political trade-offs

Change #1032888 had a related patch set uploaded (by Brian Wolff; author: Brian Wolff):

[operations/mediawiki-config@master] Allow async (job queue based) chunked upload on all wikis

https://gerrit.wikimedia.org/r/1032888

Change #1032888 merged by jenkins-bot:

[operations/mediawiki-config@master] Allow async (job queue based) chunked upload on all wikis

https://gerrit.wikimedia.org/r/1032888

Mentioned in SAL (#wikimedia-operations) [2024-05-21T08:23:46Z] <mlitn@deploy1002> Started scap: Backport for [[gerrit:1032888|Allow async (job queue based) chunked upload on all wikis (T364644)]]

Mentioned in SAL (#wikimedia-operations) [2024-05-21T08:26:26Z] <mlitn@deploy1002> mlitn and bawolff: Backport for [[gerrit:1032888|Allow async (job queue based) chunked upload on all wikis (T364644)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-05-21T08:41:19Z] <mlitn@deploy1002> Finished scap: Backport for [[gerrit:1032888|Allow async (job queue based) chunked upload on all wikis (T364644)]] (duration: 17m 32s)

Bawolff claimed this task.
This comment was removed by hinnk.

Tested on enwiki and verified that a 4GB file was now able to upload with async enabled.