Page MenuHomePhabricator

"Page cannot be displayed due to robots.txt." from ArchiveBot link
Closed, InvalidPublic

Event Timeline

Ideally the tool should be used to fix this, but it's not able to retrieve the URL, so there's clearly a bug.

Actually never mind. The URL is accessible.

https://tools.wmflabs.org/iabot/index.php?page=manageurlsingle&url=http%253A%252F%252Fwww.digital-dd.com%252Fqihoo-browser-war

You can use the tool above to change the archive above to the appropriate value.

What does this mean?

From my understanding, there is an archived page, but the archived page is inaccessible due to robot.txt.

Should we blacklist the site for the bot?

What does this mean?

From my understanding, there is an archived page, but the archived page is inaccessible due to robot.txt.

Should we blacklist the site for the bot?

The URL in the above comment takes you to my tool which lets users tell IABot what archive URL it should be using, if at all. If there aren't any viable URLs, then you can delete it and the bot won't use any archive URL. You can even alter how it considers the URL, either dead or alive.