See T127066: Bingbot scraping tools? for reason.
The current robots.txt:
User-agent: * Disallow: /betacommand-dev/ Disallow: /fiwiki-tools/ # Disallow XoviBot. User-agent: XoviBot Disallow: /
Many dynamic pages which take long time to load should be added.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Feature | None | T16720 robots.txt (tracking) | ||
| Resolved | bd808 | T127206 provide a more strict robots.txt at Tool Labs | |||
| Open | None | T226688 Block web crawlers from accessing Cloud Services |
We had a more restricted robots.txt in the past (March 2014; cf. T63132) and consensus for it. This changed in the following eleven months so when I committed d76c5f0a398b827f999fd1d8bd123798fdf0f35b, the then current version, it had become "allow by default".
IIRC my thoughts regarding robots.txt were:
Many dynamic pages which take long time to load should be added.
Do you have a representative sample of such URLs?
See T127066: Bingbot scraping tools? for reason.
That's not called a reason, it's called pre-emptive optimisation. The actual bug needs to be fixed here; pages getting visits is not a bug.
If we found that the problem is "too many concurrent requests", robots.txt should contain Crawl-delay: 1.
My tools were fine until ~Feb 14 2016. Something changed around that time; could be the DB fail, but I can't tell. Since then, some (by far not all) tools started accumulating requests and becoming non-responsive, but without the webservice actually dying.
The crawlers might just be a symptom.
OK, most tools seem to be back to normal, except "catscan2" and "glamtools". Could those be added to robots.txt please?
T251628: Serve some default well known files for Toolforge webservices has made the default /robots.txt for all tools disallow compliant crawlers. Tools that wish to be crawled can serve their own /robots.txt content. This has been announced on the cloud-announce mailing list and added to the Toolfroge webservice help on Wikitech.