Page MenuHomePhabricator

Detect robots.txt exclusion on
Closed, DeclinedPublic

Event Timeline

IABot uses filters when requesting archive copies. Any instances of defective archives must have happened retroactively.

GreenC bot detects robots.txt and will try to find a different archive if available. If none available it will keep the robots.txt snapshot, because Wayback management said they plan to remove that policy block sometime in the near future, hopefully.