Please add *.wmflabs.org to wgCopyUploadsDomains on Wikimedia Commons. This would allow bots to make the WMF server pull files instead of having the complicated and error-prone upload part on my side.
|Resolved||None||T64820 Allow copy upload files from WMF sites on Wikimedia Commons|
|Resolved||Steinsplitter||T75724 Whitelisting domain for GWToolset|
|Open||None||T60224 Add domains to $wgCopyUploadsDomains (tracking)|
|Open||None||T78167 Add *.wmflabs.org to `wgCopyUploadsDomains`|
|Resolved||akosiaris||T95714 Allow the production cluster to access *.wmflabs.org IPs|
The use case from T78167 is for wgCopyUploadsDomain:
legoktm@terbium:~$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/legobot/hi.txt curl: (56) Received HTTP code 403 from proxy after CONNECT
If I try again now, it seems to pass with:
HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/
Maybe the url-downloader did not have access to the labs reverse proxy / tools-wmflabs.org ..
T95714: Allow the production cluster to access *.wmflabs.org IPs was Resolved. Can we update this task ?
For example, can someone test on prod cluster this command ?
If it's good, we probably will be able to merge the patch that simply add the domain to the whitelist.
Yes that works, per my comment two years ago T95714#1470497 and I have confirmed it again right now:
terbium$ HTTPS_PROXY=url-downloader.wikimedia.org:8080 curl https://tools.wmflabs.org/robots.txt User-agent: * Crawl-delay: 3 ...
Most probably, url-downloader.wikimedia.org was not able to reach the wmflabs proxy.
So I guess it now it depends whether we want to allow $wgCopyUploadsDomains = '*.wmflabs.org' or a subset of subdomains or whatever. I can not tell.