@Niharika: See my comments at https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/Cyberbot_II_5a#DB_Results.

@Niharika: Do you have any examples of the pages behind paywalls?

In T132606#2205727, @kaldari wrote:

@Niharika: Do you have any examples of the pages behind paywalls?

http://www.britishnewspaperarchive.co.uk/viewer/bl/0000510/19030226/074/0006 (There are several links from this same website, all marked as dead.)

@kaldari, I saw your comments and I found multiple such URLs which return a 404/405 via curl but a 200 on browser. The reason behind this seems to be a missing User-Agent. The obvious fix is to spoof a user-agent such as

curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');

However I'm not sure of the ethical (and legal?) implications of this, if any.

You can test this as well -

curl -I -s -L http://www.oscars.org/oscars/ceremonies/1966

returns a 403 Forbidden

while...

curl -I -s -L http://www.oscars.org/oscars/ceremonies/1966 -H'User-Agent:  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:27.0) Gecko/20100101 Firefox/27.0'

returns a 200 OK

@Niharika: Thats a good idea. Let's set the User Agent string as a const in the checkIfDead class and use that with the curl requests. I don't believe there are any legal or ethical problems with spoofing user agent strings. Internet Explorer has been spoofing Netscape Navigator since 1998 :)

Strangely, this didn't fix the 404 I get for http://gym.longinestiming.com/File/000002030000FFFFFFFFFFFFFFFFFF01, but it does seem to fix others.

Done. https://github.com/cyberpower678/Cyberbot_II/pull/12

@Niharika: Needs 1 additional change. See comment at pull request.

In T132606#2210920, @kaldari wrote:

@Niharika: Needs 1 additional change. See comment at pull request.

Sorry, I missed this. https://github.com/cyberpower678/Cyberbot_II/pull/14

How dare you guys not subscribe me. :p

We should consider randomizing the UAs. We generate a list of UAs being used by legitimate browsers, and randomly pick on from the list. That should keep Cyberbot functional when scanning pages.