Change Details

The code for T122659 was a good start, but we can probably improve on it. This task it to investigate what other kinds of dead links might be detectable besides 4XX and 5XX error codes. Questions to answer: * Are there other HTTP response codes that we should consider dead links? * Is it possible to detect "soft 404s" (for example http://www.unhcr.org/home/RSDCOI/3ae6a9f444.html)? If so, how? * What about URLs that redirect to the domain root (like http://findarticles.com/p/articles/mi_m0FCL/is_4_30/ai_66760539/)? Should those be considered dead? * Is it possible to detect pages that have been replaced with link farms (sorry haven't found an example)?

The code for T122659 was a good start, but we can probably improve on it. This task it to investigate what other kinds of dead links might be detectable besides 4XX and 5XX error codes. Questions to answer: * Are there other HTTP response codes that we should consider dead links? * Is it possible to detect "soft 404s" (for example http://www.unhcr.org/home/RSDCOI/3ae6a9f444.html)? If so, how? * What about URLs that redirect to the domain root (like http://findarticles.com/p/articles/mi_m0FCL/is_4_30/ai_66760539/)? Should those be considered dead? * Is it possible to detect pages that have been replaced with link farms (sorry haven't found an example)? * How thorough is [[ https://www.mediawiki.org/wiki/Manual:Pywikibot/weblinkchecker.py | weblinkchecker]]'s dead link checking? Should we try to use that instead (or port some of it's logic to the Cyberbot code)?

The code for T122659 was a good start, but we can probably improve on it. This task it to investigate what other kinds of dead links might be detectable besides 4XX and 5XX error codes. Questions to answer: * Are there other HTTP response codes that we should consider dead links? * Is it possible to detect "soft 404s" (for example http://www.unhcr.org/home/RSDCOI/3ae6a9f444.html)? If so, how? * What about URLs that redirect to the domain root (like http://findarticles.com/p/articles/mi_m0FCL/is_4_30/ai_66760539/)? Should those be considered dead? * Is it possible to detect pages that have been replaced with link farms (sorry haven't found an example)? * How thorough is [[ https://www.mediawiki.org/wiki/Manual:Pywikibot/weblinkchecker.py | weblinkchecker]]'s dead link checking? Should we try to use that instead (or port some of it's logic to the Cyberbot code)?