Page MenuHomePhabricator

Please add www.defenceimagery.mod.uk to $wgCopyUploadsDomains
Closed, ResolvedPublic

Description

For several years I have uploaded high quality photographs from the UK Ministry of Defence database and would like to move to url uploads rather than local caches. The upload links after a header redirect look like www.defenceimagery.mod.uk/fotoweb/cmdrequest/rest/Download.fwx/45153802.jpg - along with an auto-generated session cookie parameter.

I'm guessing that white-listing www.defenceimagery.mod.uk would be sufficient.

Example past images: Here

Event Timeline

Fae created this task.May 24 2017, 10:05 PM
Restricted Application added subscribers: Bawolff, Aklapper. · View Herald TranscriptMay 24 2017, 10:05 PM
Framawiki triaged this task as Normal priority.May 25 2017, 7:42 AM
Framawiki claimed this task.
Framawiki added a subscriber: Framawiki.

Of what I could test, the cookie is mandatory. I don't know if it'll works, but let's try.

Change 355594 had a related patch set uploaded (by Framawiki; owner: Framawiki):
[operations/mediawiki-config@master] Add www.defenceimagery.mod.uk to CopyUploadsDomains

https://gerrit.wikimedia.org/r/355594

Dereckson added a subscriber: Dereckson.

@Fae would you have an URL example so we can test how the access token behaves?

Of what I could test, the cookie is mandatory. I don't know if it'll works, but let's try.

We can test now with a token generateed by another user.

It's look difficult to having these files with curl and cookies jars. Perhaps we can just ask them to allow our ips ?

Framawiki removed Framawiki as the assignee of this task.May 25 2017, 10:24 PM
Fae added a comment.EditedMay 30 2017, 10:04 PM

I'm unclear as to why we are worried about tokens. If url upload is allowed, then the URL with a token passed as a parameter looks like:
http://www.defenceimagery.mod.uk/fotoweb/cmdrequest/rest/Download.fwx/45153802.jpg?D=EFCC51FEE65DA414D18085DA188CAB45524FFC4F7A63A403C47E17A8BEF1E554B796D6EA4FD91784A04B36049843E1FB56B129047A099FD2448D5AA2FD3EBB84D49852E5EF22F9F1E9930FDF2671F90028F4747E4DAAD3BE496BC62277DF33E1BC24AB66E7B4B90225B163F54F224DFE65DFE22A5F65B6D1328840103D2F128F615EE150C8AA32E00FC8DA1E13BEA266&ForceSaveDialog=no

I believe the system can upload from this address, so long as the domain is whitelisted. Naturally, the Commons image page would give a generic link to the source, rather than include session specific tokens.

This link works for me. @Dereckson are you agree to add this domain ?

If system sees the URL (tokens are valid worldwide), then it should be able
to upload. For me, link works.

If creating URL list is possible, why not. If it isn't, they must upload
images to other location anyway, so notify and whitelist.

Anyway, why we actually whitelist? Is there a Dos attack possible? Only
from our infrastructure i think but this can be solved by throttling upload
by URL actions by user. In some time maybe autoblock them.

Fae added a comment.Jun 3 2017, 8:48 PM

...

Anyway, why we actually whitelist? Is there a Dos attack possible? Only
from our infrastructure i think but this can be solved by throttling upload
by URL actions by user. In some time maybe autoblock them.

This is an interesting point. Would it be worth creating a task to look at creating a user-whitelist for url uploads, where users appropriately experienced in batch uploads can be granted the right to run upload projects for any source?

It may be a solution. But I think much better would be simply give the upload from any source right to all current holders of upload_by_url right (no matter if we create separate upload_by_any_url and give it to all holders of upload_by_url by WMF config or simply remove the whitelist by conf variable). Why it can be destructive?

I'll deploy this in one hour, morning swat, since it seems to work

Change 355594 merged by jenkins-bot:
[operations/mediawiki-config@master] Add www.defenceimagery.mod.uk to CopyUploadsDomains

https://gerrit.wikimedia.org/r/355594

Framawiki claimed this task.Jun 5 2017, 7:30 PM

@Fae Deployed, can you check that it works ?

Framawiki closed this task as Resolved.Jun 5 2017, 8:18 PM

Good news !