mdale wrote:

Would like to bump this up now that its better supported and running successfully running on testing.wikimedia.org.

November 5, «15:13 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling upload by URL for sysops on all wikis'».
December 1, «15:20 apergos: temporarily turned off upload-by-url on test.wikipedia, it interferes with testing the external page retrieval extension for fundraising».
We still have upload_by_url right on all wikis, but actual upload by URL is not enabled.

mdale wrote:

Any update on turning this back on?

Enabled on test and commons; tested it and it works on both.

Disabled again, breaks Lucene search.

mdale wrote:

Fix committed in r60811 ... this re-merges new HttpFunctions.php from js2 back into trunk. Its much closer to what is currently in the deployment branch, and is needed to support async copy-by-url uploads.

Tested locally with copy-by-url uploads. Hard to test the Lucene setup outside of the wikimedia.

Reopening, the request itself (enable upload by URL) hasn't been fulfilled.

mdale wrote:

Could we at least enable $wgAllowCopyUploads without background uploading?

mdale wrote:

Bryon has been working on a jobQueue implementation.

Bryon do you have any updates on how well its supported in trunk presently and if there are any outstanding issues of this feature that your aware of?

Bryan.TongMinh wrote:

As far as I'm concerned all outstanding issues have been fixes.

Also, there is both a Brion and a Bryan. There is no such thing as a Bryon though.

mdale wrote:

Sorry "Bryan". Great to hear the outstanding issues have been fixed :) Can we add this to the list of feature to deploy soon after we review and deploy trunk? Is there a bug or road map page tracking that somewhere Roan, Rob ?

mdale wrote:

Flagging this bug for additional comment, now that the dust has settled from 1_17 it would be great if we could review setting $wgAllowCopyUploads to true for commons.

mdale wrote:

Any update on enabling this?

mdale wrote:

Any update on enabling this?

(In reply to comment #14)

Any update on enabling this?

Seems we've got some infrastructure issues as the apaches don't have direct external internet access...

Need to discuss with Tim

Please see bug 14919 comment 5 - I didn't notice this followup bug.

Needs review.

• Bug 14919 has been marked as a duplicate of this bug. ***

• Bug 1552 has been marked as a duplicate of this bug. ***

• Bug 5283 has been marked as a duplicate of this bug. ***

• Bug 1552 has been marked as a duplicate of this bug. ***

• Bug 35263 has been marked as a duplicate of this bug. ***

M8R-udfkkf wrote:

Any progress/updates?

• Bug 37636 has been marked as a duplicate of this bug. ***

Any progress/updates?

We now have a custom Flickr uploading interface that was built as a GSoC project: https://gerrit.wikimedia.org/r/#/c/12269/

But unfortunately it depends on $wgAllowCopyUploads. It would be really nice to have this available during Wiki Loves Monuments.

Any update on getting internet access to the apaches?

According to Tim, there is already a proxy set up called url-downloader.wikimedia.org which should allow the apaches to access the outside world. He also said that UploadFromUrl.php could use a good review, especially for error reporting. Other than that, I don't think there are any blockers for turning this back on.

sumanah wrote:

Ryan, maybe you could file an RT ticket to get this higher on ops's priority list? Thanks.

I don't think there's necessarily any ops work involved. We need to review the upload from URL code and verify whether there's still any Lucene breakage in production; the URL downloader itself is operational according to Mark.

Unfortunately I don't know enough about Lucene to tell what would cause problems. The impression I got from Tim is that the search issue was no longer a problem, but it would be best to get confirmation from him directly or someone familiar with Lucene.

Oren?

(In reply to comment #30)

Unfortunately I don't know enough about Lucene to tell what would cause
problems. The impression I got from Tim is that the search issue was no longer
a problem, but it would be best to get confirmation from him directly or
someone familiar with Lucene.

The Lucene issue was fixed with a hack in the MWSearch extension, but setting $wgHTTPProxy would still break any other Http::get() caller that is attempting to fetch things from the local intranet. I think the configuration should be split: UploadFromUrl should pass a "proxy" option to MWHttpRequest::factory() with a value derived from a global variable specific to this feature.

Could we bump up the priority of this? I am surprised it is still not working on Wikimedia Commons.

(In reply to comment #32)

(In reply to comment #30)

Unfortunately I don't know enough about Lucene to tell what would cause
problems. The impression I got from Tim is that the search issue was no longer
a problem, but it would be best to get confirmation from him directly or
someone familiar with Lucene.

The Lucene issue was fixed with a hack in the MWSearch extension, but setting
$wgHTTPProxy would still break any other Http::get() caller that is attempting
to fetch things from the local intranet. I think the configuration should be
split: UploadFromUrl should pass a "proxy" option to MWHttpRequest::factory()
with a value derived from a global variable specific to this feature.

https://gerrit.wikimedia.org/r/#/c/25605/

Enabled on testwiki

Uploads from HTTPS sites fail with an error of:
Error fetching URL: Received HTTP code 403 from proxy after CONNECT

I tried it using a script (via API), but it returned HTTP 500 when trying to print the error message :(

see http://pastebin.com/raw.php?i=NXLPRteT

Looks like uploads from regular HTTP sites also return 403 Forbidden.

Well, actually I can upload from http://farm1.staticflickr.com, but not http://upload.wikimedia.org

Http worked before, as I did the responsible thing and imported copyrighted Google logos. Was a couple of weeks ago now.

Might need an exception in the squid config to do local requests..

We *need* this for this migration of Wiki Voyage/Travel to Wikimedia. There are 34,124 files to upload.

No we don't. We can bulk upload them using a maintenance script

We did need it for WLM, but that was 2 weeks ago :(

Since Flickr uploading seems to work, would anyone object to me moving forward with Flickr-only uploading for admins? This would involve turning on $wgAllowCopyUploads, turning off $wgCopyUploadsFromSpecialUpload, setting $wgCopyUploadsDomains to array( '*.flickr.com' ), finishing up https://gerrit.wikimedia.org/r/#/c/12269/ for UploadWizard, and setting $wgGroupPermissions['sysop']['upload_by_url'] to true.

(In reply to comment #42)

We did need it for WLM, but that was 2 weeks ago :(

Since Flickr uploading seems to work, would anyone object to me moving forward
with Flickr-only uploading for admins? This would involve turning on
$wgAllowCopyUploads, turning off $wgCopyUploadsFromSpecialUpload, setting
$wgCopyUploadsDomains to array( '*.flickr.com' ), finishing up
https://gerrit.wikimedia.org/r/#/c/12269/ for UploadWizard, and setting
$wgGroupPermissions['sysop']['upload_by_url'] to true.

Seems sane to me, proxy should be alright for now...

Reading through the comments, I don't see any pending ops work for this, so I'm removing the ops tag. If I'm mistaken, please re-add and clarify what needs to be done from our side.

rd232 wrote:

Ping! Commons would love to have this, as discussed again at http://commons.wikimedia.org/wiki/Commons:Village_pump/Proposals#Upload_by_URL

Stay tuned...

Looks like Gerrit change 34357 enables this with flickr domains for commons admins.

sumanah wrote:

Gerrit change #34357 is now merged and https://gerrit.wikimedia.org/r/#/c/25605/ is merged; ok to close this ticket?

This isn't working at the moment, because it is apparently "disabled until author bug is fixed". Is this "author bug" tracked?

(In reply to comment #49)

This isn't working at the moment, because it is apparently "disabled until
author bug is fixed".

aka Ie59c3eb7

Is this "author bug" tracked?

Yes, bug 42312, and it's fixed. It's useful if you test it on [[testwiki:]] and mark it VERIFIED if ok.

As soon as https://gerrit.wikimedia.org/r/#/c/34856/ is merged and deployed, I'll turn $wgAllowCopyUploads on and we can close this bug. We still need to file a separate bug for HTTPS support for the proxy service, but that isn't a blocker for enabling $wgAllowCopyUploads.

Also note that when $wgAllowCopyUploads is turned on, it's only going to be available in a very limited sense: for admins on Commons, via UploadWizard, from Flickr. People may want to file new bugs to change any of these individual limitations.

$wgAllowCopyUploads turned on for Commons in https://gerrit.wikimedia.org/r/#/c/37353/

This feature is currently limited to administrators, via UploadWizard, from Flickr.com.

Feel free to file new bugs if you'd like any of these restrictions changed.

Upload from Flickr works for Firefox on Windows XP, but not with Chrome Version 23.0.1271.10. Should I open a new bug?

Yes, please file a bug for that. BTW, some people have had trouble using Flickr uploading if they have the HTTPS Everywhere extension turned on, so you may want to check that.

https://bugzilla.wikimedia.org/show_bug.cgi?id=43422