guc is being scraped by OrangeBot, Googlebot/2.1, Baiduspider/2.0 and Yeti/1.1 at the same time. Even with reasonable delays between requests (3s), this effectively brings guc down.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| robots.txt: disallow /guc/ | labs/toollabs | master | +3 -0 |
Related Objects
Related Objects
Event Timeline
Comment Actions
Change 292381 had a related patch set uploaded (by Merlijn van Deen):
robots.txt: disallow /guc/
Comment Actions
- robots.txt change reverted in dca1dc208aa1d42cf30157f1a93c75005f61954f.
- rTGUCd22c41f5b283: Don't index username queries - https://github.com/wikimedia/labs-tools-guc/commit/d22c41f5b2
- rTGUC9e4d72ad5c9a: Try to avoid auto-submit for bots or spiders on GET - https://github.com/wikimedia/labs-tools-guc/commit/9e4d72ad5c