Page MenuHomePhabricator
Create Task
Maniphest T15307

Robots.txt exempt for browsershots
Closed, DeclinedPublic
Actions

Description

Hiya,

Whenever someone gets a chance, pretty please add an exempt for browsershots to robots.txt on enwiki (based on http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29/Archive_18#robots.txt_.2F_browsershots as well as http://en.wikipedia.org/wiki/User_talk:Slakr#Browsershots)

E.g.,:

User-agent: Browsershots
Disallow:

... or something similar. That would help tremendously in examining cross-compatibility between browsers and platforms.

Thanks a million, and cheers =)
--slakr@enwiki

Version: unspecified
Severity: enhancement
URL: http://en.wikipedia.org/robots.txt

Details

Reference
bz13307

Event Timeline

• bzimport raised the priority of this task from to Lowest.Nov 21 2014, 10:05 PM
• bzimport added projects: Wikimedia-Site-requests, Shell.
• bzimport set Reference to bz13307.
• bzimport added a subscriber: Unknown Object (MLST).
Slakr created this task.Mar 10 2008, 4:39 AM
• brion added a comment.Mar 10 2008, 7:03 PM

There shouldn't be anything blocking Browsershots from accessing pages; certainly nothing apparent in our robots.txt

From my previous testing, it appears that Browsershots itself is very badly configured:

  1. It first loads /robots.txt with a generic "Python-urllib" user-agent. This is blocked at the HTTP proxy level on Wikimedia sites due to past abuse, so they would be unable to load our robots.txt file.
  1. It then loads the requested page with an app-specific "Browsershots" user-agent. This is not blocked, and would be allowed with no problems.
  1. It then passes the page off to a bunch of browsers in turn.

I can only assume that the failure to read robots.txt is interpreted as a bad sign. :)

Please contact Browsershots and tell them that their bot is broken; they should load robots.txt with the same user-agent string that they load the page itself with.

• bzimport added a comment.Apr 16 2008, 5:49 AM

jeluf wrote:

Closed.

• MZMcBride added a comment.Mar 11 2011, 12:57 AM
  • Bug 27986 has been marked as a duplicate of this bug. ***
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL