Page MenuHomePhabricator

Block spider / web crawler on tool labs
Closed, InvalidPublic


Author: metatron

Tracking ticket to block aggressive spiders and web crawlers on tool labs (*)

These spiders should be blocked at network or proxy level rather than in individual lighttpd-configs or even applications to avoid a waste of resources.

Version: unspecified
Severity: normal



Related Objects


Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:33 AM
bzimport set Reference to bz68300.

metatron wrote:

Aggressive species:

-Sogou web spider
-Yahoo! Slurp

User agents:
"Mozilla/5.0 (compatible; SeznamBot/3.2; +"
"Mozilla/5.0 (compatible; SputnikBot/2.3; +
"Sogou web spider/4.0(+"
"Mozilla/5.0 (compatible; TweetmemeBot/3.0; +"
"kinshoobot (/global; amd64 Linux 3.10.23-xxxx-std-ipv6-64; java 1.8.0_05; Europe/fr)"
"CCBot/2.0 ("
"Scrapy/0.22.0 (+"
"Mozilla/5.0 (compatible; Baiduspider/2.0; +"
"Mozilla/5.0 (compatible; Yahoo! Slurp;"

metatron wrote:

  • 360Spider

User Agent:
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider"

Are these still a problem? The proxy can now reject based on User Agents, so these should be trivial to implement. TweetmemeBot is already blocked.

A tracking task with only one (resolved) blocking task doesn't make much sense to me :-).

There are certainly some things to improve about our spider treatment, and one of them is probably that we shouldn't deal with this case by case. However, due to the syntax of robots.txt, IIRC allowing bots to access / and the status pages below requires explicitly disallowing all other paths. So we would need to generate robots.txt dynamically by nginx including all tools except for some whitelist where the tool maintainer said: "I guarantee that bots can spider this tool without performance impact."

This sounds like work to be coordinated with a lot of people, and I would procrastinate that until the pain level rises.

Danny_B renamed this task from [tracking] Block spider / web crawler on tool labs to Block spider / web crawler on tool labs.Jul 29 2016, 4:25 PM