Page MenuHomePhabricator

Add *.wmflabs.org to `wgCopyUploadsDomains`
Open, NormalPublic

Description

Please add *.wmflabs.org to wgCopyUploadsDomains on Wikimedia Commons. This would allow bots to make the WMF server pull files instead of having the complicated and error-prone upload part on my side.

Event Timeline

Rillke created this task.Dec 10 2014, 5:53 PM
Rillke updated the task description. (Show Details)
Rillke raised the priority of this task from to Needs Triage.
Rillke changed Security from none to None.
Rillke added a subscriber: Rillke.
Rillke updated the task description. (Show Details)Dec 10 2014, 5:56 PM

Change 179094 had a related patch set uploaded (by Steinsplitter):
Adding *.wmflabs.org to wgCopyUploadsDomains

https://gerrit.wikimedia.org/r/179094

Patch-For-Review

Another advantage is that production servers would decide when they want to fetch the file.

Steinsplitter triaged this task as Normal priority.Dec 11 2014, 11:57 AM
Steinsplitter claimed this task.
Glaisher edited projects, added Shell; removed acl*sre-team.Dec 13 2014, 6:13 AM

Last I checked, production servers can't talk to labs, so even if you add the domain, I'm pretty sure it won't work.

Reedy added a subscriber: Reedy.Dec 13 2014, 10:28 PM

If someone has a url of a file hosted on labs, we can easily confirm...

legoktm@terbium:~$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/legobot/hi.txt
curl: (56) Received HTTP code 403 from proxy after CONNECT

production servers can't talk to labs

And what is the reason for that? The same invalid one as in T44473 ?

Change 179094 abandoned by Steinsplitter:
Adding *.wmflabs.org to wgCopyUploadsDomains

Reason:
see T78167, wmflabs in internal, so it dosen't work

https://gerrit.wikimedia.org/r/179094

Steinsplitter removed Steinsplitter as the assignee of this task.Jan 24 2015, 7:33 PM
tomasz removed a project: Shell.Feb 23 2015, 7:56 PM
Steinsplitter moved this task from Incoming to Uploading on the Commons board.
Dzahn changed the task status from Open to Stalled.May 7 2015, 12:16 AM
Dzahn added a subscriber: Dzahn.
Steinsplitter changed the task status from Stalled to Open.May 19 2015, 6:03 PM
Yann added a subscriber: Yann.May 19 2015, 6:57 PM

Hi, Could you speed up this task a bit please? It would be quite useful for many tools on the Labs (e.g. https://tools.wmflabs.org/yifeibot/gallica.py). It is not really a great deal to reconfigure a proxy...

You realise this task has an open blocker, right?

Yann added a comment.May 19 2015, 7:58 PM

Yes, I added a word there.

You realise this task has an open blocker, right?

Somone schould work on the "blocker"....

hashar added a subscriber: hashar.Jul 22 2015, 12:03 PM

The use case from T78167 is for wgCopyUploadsDomain:

legoktm@terbium:~$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/legobot/hi.txt
curl: (56) Received HTTP code 403 from proxy after CONNECT

If I try again now, it seems to pass with:

HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/

Maybe the url-downloader did not have access to the labs reverse proxy / tools-wmflabs.org ..

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 22 2015, 12:03 PM
Dereckson added a subscriber: csteipp.EditedFeb 11 2016, 4:24 AM

So let's summarize.

  1. T95714 is marked as declined.
  2. @csteipp expressed several times an opinion we should prepare exceptions one per one, and not allow *.wmflabs.org

Would someone see any solution to whitelist a domain we can't access or could we resolve as WONTFIX this task too?

Restricted Application added a subscriber: JEumerus. · View Herald TranscriptFeb 11 2016, 4:24 AM
Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptApr 14 2016, 6:50 PM
  1. T95714 is marked as declined.
  2. @csteipp expressed several times an opinion we should prepare exceptions one per one, and not allow *.wmflabs.org

    Would someone see any solution to whitelist a domain we can't access or could we resolve as WONTFIX this task too?

I guess that means declining this task. :-/

Dereckson closed this task as Declined.May 4 2016, 10:12 AM

So we're declining the task, with the precision it could be allowed in the future to allow <subdomain>.wmflabs.org.

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptMay 4 2016, 10:12 AM
Multichill reopened this task as Open.May 4 2016, 10:33 AM

So we're declining the task, with the precision it could be allowed in the future to allow <subdomain>.wmflabs.org.

The blocking task (T95714) was also reopened so we shouldn't close this one.

Dzahn removed a subscriber: Dzahn.May 4 2016, 1:30 PM

T95714: Allow the production cluster to access *.wmflabs.org IPs was Resolved. Can we update this task ?

For example, can someone test on prod cluster this command ?

legoktm@terbium:~$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/robots.txt

If it's good, we probably will be able to merge the patch that simply add the domain to the whitelist.

T95714: Allow the production cluster to access *.wmflabs.org IPs was Resolved. Can we update this task ?

For example, can someone test on prod cluster this command ?

legoktm@terbium:~$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/robots.txt

If it's good, we probably will be able to merge the patch that simply add the domain to the whitelist.

Yes that works, per my comment two years ago T95714#1470497 and I have confirmed it again right now:

terbium$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/robots.txt
User-agent: *
Crawl-delay: 3
...

Most probably, url-downloader.wikimedia.org was not able to reach the wmflabs proxy.


So I guess it now it depends whether we want to allow $wgCopyUploadsDomains = '*.wmflabs.org' or a subset of subdomains or whatever. I can not tell.