Add *.toolforge.org to `wgCopyUploadsDomains`
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	Rillke
	Dec 10 2014, 5:53 PM

Description

Please add *.wmflabs.org to wgCopyUploadsDomains on Wikimedia Commons. This would allow bots to make the WMF server pull files instead of having the complicated and error-prone upload part on my side.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	commonswiki: Add *.toolforge.org to wgCopyUploadsDomains allowlist	operations/mediawiki-config	master	+1 -2
	Adding *.wmflabs.org to wgCopyUploadsDomains	operations/mediawiki-config	master	+1 -0

Customize query in gerrit

Related Objects
Search...

View Standalone Graph

This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Status	Assigned	Task
		· · ·
Resolved	None	T60224 Add domains to $wgCopyUploadsDomains (tracking)
Declined	None	T78167 Add *.toolforge.org to `wgCopyUploadsDomains`
Resolved	akosiaris	T95714 Allow the production cluster to access *.wmflabs.org IPs
		· · ·

Event Timeline

Rillke created this task.Dec 10 2014, 5:53 PM

Rillke raised the priority of this task from to Needs Triage.

Rillke updated the task description. (Show Details)

Rillke added projects: acl*sre-team, Wikimedia-Site-requests.

Rillke changed Security from none to None.

Rillke added subscribers: Steinsplitter, JeanFred, Multichill, dschwen.

Rillke subscribed.

Rillke updated the task description. (Show Details)Dec 10 2014, 5:56 PM

Change 179094 had a related patch set uploaded (by Steinsplitter):
Adding *.wmflabs.org to wgCopyUploadsDomains

https://gerrit.wikimedia.org/r/179094

Patch-For-Review

Another advantage is that production servers would decide when they want to fetch the file.

Steinsplitter claimed this task.Dec 11 2014, 11:57 AM

Steinsplitter triaged this task as Medium priority.

Glaisher edited projects, added Shell; removed acl*sre-team.Dec 13 2014, 6:13 AM

Last I checked, production servers can't talk to labs, so even if you add the domain, I'm pretty sure it won't work.

If someone has a url of a file hosted on labs, we can easily confirm...

legoktm@terbium:~$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/legobot/hi.txt
curl: (56) Received HTTP code 403 from proxy after CONNECT

production servers can't talk to labs

And what is the reason for that? The same invalid one as in T44473 ?

Change 179094 abandoned by Steinsplitter:
Adding *.wmflabs.org to wgCopyUploadsDomains

Reason:
see T78167, wmflabs in internal, so it dosen't work

https://gerrit.wikimedia.org/r/179094

Steinsplitter removed a project: Patch-For-Review.Dec 18 2014, 5:55 PM

Steinsplitter removed Steinsplitter as the assignee of this task.Jan 24 2015, 7:33 PM

Liuxinyu970226 subscribed.Feb 19 2015, 12:35 AM

tomasz removed a project: Shell.Feb 23 2015, 7:56 PM

Steinsplitter added a project: Commons.Mar 30 2015, 4:26 PM

Steinsplitter moved this task from Incoming to Uploading on the Commons board.

Steinsplitter mentioned this in T44473: Allow upload-by-URL from upload.wikimedia.org.Apr 10 2015, 2:35 PM

Dereckson added a project: Blocked-on-Operations.Apr 10 2015, 4:22 PM

Dereckson mentioned this in T95714: Allow the production cluster to access *.wmflabs.org IPs.

Dereckson subscribed.

Dereckson moved this task from Backlog to Analysis / under discussion on the Wikimedia-Site-requests board.Apr 10 2015, 4:27 PM

Dzahn changed the task status from Open to Stalled.May 7 2015, 12:16 AM

Dzahn subscribed.

Steinsplitter changed the task status from Stalled to Open.May 19 2015, 6:03 PM

Hi, Could you speed up this task a bit please? It would be quite useful for many tools on the Labs (e.g. https://tools.wmflabs.org/yifeibot/gallica.py). It is not really a great deal to reconfigure a proxy...

You realise this task has an open blocker, right?

Yes, I added a word there.

In T78167#1297229, @Krenair wrote:

You realise this task has an open blocker, right?

Somone schould work on the "blocker"....

zhuyifei1999 subscribed.May 26 2015, 3:01 PM

faidon removed a project: Blocked-on-Operations.May 27 2015, 5:13 PM

zhuyifei1999 added a parent task: T60224: Add domains to $wgCopyUploadsDomains (tracking).Jul 15 2015, 11:45 AM

The use case from T78167 is for wgCopyUploadsDomain:

legoktm@terbium:~$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/legobot/hi.txt
curl: (56) Received HTTP code 403 from proxy after CONNECT

If I try again now, it seems to pass with:

HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/

Maybe the url-downloader did not have access to the labs reverse proxy / tools-wmflabs.org ..

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 22 2015, 12:03 PM

Matanya closed subtask T95714: Allow the production cluster to access *.wmflabs.org IPs as Declined.Jul 22 2015, 12:08 PM

zhuyifei1999 moved this task from Uploading to Uploading (upload_by_url and server side upload) on the Commons board.Nov 15 2015, 8:13 AM

So let's summarize.

T95714 is marked as declined.
@csteipp expressed several times an opinion we should prepare exceptions one per one, and not allow *.wmflabs.org

Would someone see any solution to whitelist a domain we can't access or could we resolve as WONTFIX this task too?

Restricted Application added a subscriber: JEumerus. · View Herald TranscriptFeb 11 2016, 4:24 AM

Andrew reopened subtask T95714: Allow the production cluster to access *.wmflabs.org IPs as Open.Apr 14 2016, 6:50 PM

Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptApr 14 2016, 6:50 PM

In T78167#2017793, @Dereckson wrote:

T95714 is marked as declined.

@csteipp expressed several times an opinion we should prepare exceptions one per one, and not allow *.wmflabs.org

Would someone see any solution to whitelist a domain we can't access or could we resolve as WONTFIX this task too?

I guess that means declining this task. :-/

So we're declining the task, with the precision it could be allowed in the future to allow <subdomain>.wmflabs.org.

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptMay 4 2016, 10:12 AM

In T78167#2263074, @Dereckson wrote:

So we're declining the task, with the precision it could be allowed in the future to allow <subdomain>.wmflabs.org.

The blocking task (T95714) was also reopened so we shouldn't close this one.

Dzahn unsubscribed.May 4 2016, 1:30 PM

akosiaris closed subtask T95714: Allow the production cluster to access *.wmflabs.org IPs as Resolved.Jan 23 2017, 1:48 PM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:56 PM

T95714: Allow the production cluster to access *.wmflabs.org IPs was Resolved. Can we update this task ?

For example, can someone test on prod cluster this command ?

In T78167#846423, @Legoktm wrote:

legoktm@terbium:~$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/robots.txt

If it's good, we probably will be able to merge the patch that simply add the domain to the whitelist.

In T78167#3419909, @Framawiki wrote:
T95714: Allow the production cluster to access *.wmflabs.org IPs was Resolved. Can we update this task ?

For example, can someone test on prod cluster this command ?
In T78167#846423, @Legoktm wrote:
legoktm@terbium:~$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/robots.txt
If it's good, we probably will be able to merge the patch that simply add the domain to the whitelist.

Yes that works, per my comment two years ago T95714#1470497 and I have confirmed it again right now:

terbium$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/robots.txt
User-agent: *
Crawl-delay: 3
...

Most probably, url-downloader.wikimedia.org was not able to reach the wmflabs proxy.

So I guess it now it depends whether we want to allow $wgCopyUploadsDomains = '*.wmflabs.org' or a subset of subdomains or whatever. I can not tell.

taavi mentioned this in T255363: Add parliamentdiagram.toolforge.org to the wgCopyUploadsDomains whitelist of Wikimedia Commons.Jun 15 2020, 8:02 AM

Hi. With the blocker task being resolved, how shall this task move? Thanks.

Stang subscribed.May 8 2022, 5:58 AM

Stang added a comment.May 10 2022, 11:46 PM

This comment was removed by Stang.

Based on T292213 (and also T255363), I tried to upload a file from https://wikisource-bot.toolforge.org and it looks good, and I believe there should be no technical barrier from implementing this. Also as stated in T95714#1303192, we should trust those (toolforge user).