Change Details

Some websites, like YouTube, OpenStreetMap and Google Maps, can't be archived using the Internet Archive because of their use of JavaScript or other technology. Others, like [[ https://web.archive.org/save/https://www.hyd.gov.hk/ | large parts of the Hong Kong government's website ]], actively or passively refuse connections from the Internet Archive's Save Page Now tool; others can't be archived because of their robots.txt; and still others, like the //New York Times//, deliberately show an error message even if the content has been loaded to prevent their paywall from being circumvented. Increasing the frequency of the Internet Archive's IRC bot (T199193) is not as helpful if it's still unable to archive a lot of pages. It would be really, really nice if the process of archiving links (regardless of the entity running it) could have fallback methods for archive attempts that don't work. (I know the Internet Archive does have a YouTube collection, but it's fairly difficult to actually get content into it, it seems to be only for copyleft/public domain videos, and I don't think the bot is capable of adding to it.) [[ https://en.wikipedia.org/wiki/Help:Archiving_a_source | w:en:Help:Archiving a source ]] lists five web archiving services, including the Internet Archive. All five of them have drawbacks which prevent them from being used to archive everything. * Archive.is has a limit of about 30 pages per minute, and actively tries to prevent mass archival. I know this because they've probably blocked my ISP because I tried to archive too many pages (don't worry about it; it only happened after tens of thousands of pages). However, it has several fairly good fallback mechanisms to archive pages, including the Google cache and some other websites. It's also permanently logged in to a Facebook account. * Webcitation.org is fairly bad at archiving CSS, in my experience, and [[ https://www.webcitation.org/6zExGYP3x | also fails ]] on some Hong Kong government websites. It doesn't really seem to have any benefits over web.archive.org. * Perma.cc is only used in 62 English Wikipedia articles as of this writing. I don't know much about it but it has a limit of 10 pages per month per account. Perhaps @Green_Cardamom might know more. * Webrecorder is only used in 16 articles as of this writing. It's a nonprofit which was created this year, and as far as I'm aware is the only archiving service of the five which can save [[ https://webrecorder.io/wikipedia_1/youtube | YouTube videos ]]* (try loading the comments – you'll get to exactly where I stopped scrolling), [[ https://webrecorder.io/wikipedia_1/openstreetmap/20180716135655/https://www.openstreetmap.org/#map=14/51.5029/-0.1225 | OpenStreetMap's "slippy map" ]], and other interactive/dynamic content. It also has virtual machines, presumably for viewing outdated HTML. However, it would be more difficult to fix dead links with it because all archive links contain the name of the account which was used to create them (i.e. URLs can't automatically redirect like they do at web.archive.org). It would also probably be fairly difficult to crawl sites without support from the site owners. Having a ninety-something percent archival success rate is still not really ideal if it's out of millions of links per year, many of which will be gone within a few months of their addition to articles. The main reason I'm posting this here is that [[ https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(idea_lab)#Running_our_own_archive_service? | there is currently a Village pump (idea lab) ]] thread on the English Wikipedia proposing that the WMF runs their own archiving service. While I personally think this is somewhat unnecessary and would be an inappropriate use of WMF resources, especially when there are other non-profits which are already dedicated to web archiving, the current options are not satisfactory for archiving everything that should be archived. (* – the YouTube archival doesn't totally work because YouTube's links generate some sort of error and not all of the video quality settings can be chosen)

Some websites, like YouTube, OpenStreetMap and Google Maps, can't be archived using the Internet Archive because of their use of JavaScript or other technology. Others, like [[ https://web.archive.org/save/https://www.hyd.gov.hk/ | large parts of the Hong Kong government's website ]], actively or passively refuse connections from the Internet Archive's Save Page Now tool; others can't be archived because of their robots.txt; and still others, like the //New York Times//, deliberately show an error message even if the content has been loaded to prevent their paywall from being circumvented. Increasing the frequency of the Internet Archive's IRC bot (T199193) is not as helpful if it's still unable to archive a lot of pages. It would be really, really nice if the process of archiving links (regardless of the entity running it) could have fallback methods for archive attempts that don't work. (I know the Internet Archive does have a YouTube collection, but it's fairly difficult to actually get content into it, it seems to be only for copyleft/public domain videos, and I don't think the bot is capable of adding to it.) [[ https://en.wikipedia.org/wiki/Help:Archiving_a_source | w:en:Help:Archiving a source ]] lists five web archiving services, including the Internet Archive. All five of them have drawbacks which prevent them from being used to archive everything. * Archive.is has a limit of about 30 pages per minute, and actively tries to prevent mass archival. I know this because they've probably blocked my ISP because I tried to archive too many pages (don't worry about it; it only happened after tens of thousands of pages). However, it has several fairly good fallback mechanisms to archive pages, including the Google cache and some other websites. It's also permanently logged in to a Facebook account, and is the only one of the five to have a Tor hidden service. * Webcitation.org is fairly bad at archiving CSS, in my experience, and [[ https://www.webcitation.org/6zExGYP3x | also fails ]] on some Hong Kong government websites. It doesn't really seem to have any benefits over web.archive.org. * Perma.cc is only used in 62 English Wikipedia articles as of this writing. I don't know much about it but it has a limit of 10 pages per month per account. Perhaps @Green_Cardamom might know more. * Webrecorder is only used in 16 articles as of this writing. It's a nonprofit which was created this year, and as far as I'm aware is the only archiving service of the five which can save [[ https://webrecorder.io/wikipedia_1/youtube | YouTube videos ]]* (try loading the comments – you'll get to exactly where I stopped scrolling), [[ https://webrecorder.io/wikipedia_1/openstreetmap/20180716135655/https://www.openstreetmap.org/#map=14/51.5029/-0.1225 | OpenStreetMap's "slippy map" ]], and other interactive/dynamic content. It also has virtual machines, presumably for viewing outdated HTML. However, it would be more difficult to fix dead links with it because all archive links contain the name of the account which was used to create them (i.e. URLs can't automatically redirect like they do at web.archive.org). It would also probably be fairly difficult to crawl sites without support from the site owners. Having a ninety-something percent archival success rate is still not really ideal if it's out of millions of links per year, many of which will be gone within a few months of their addition to articles. The main reason I'm posting this here is that [[ https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(idea_lab)#Running_our_own_archive_service? | there is currently a Village pump (idea lab) ]] thread on the English Wikipedia proposing that the WMF runs their own archiving service. While I personally think this is somewhat unnecessary and would be an inappropriate use of WMF resources, especially when there are other non-profits which are already dedicated to web archiving, the current options are not satisfactory for archiving everything that should be archived. (* – the YouTube archival doesn't totally work because YouTube's links generate some sort of error and not all of the video quality settings can be chosen)

Some websites, like YouTube, OpenStreetMap and Google Maps, can't be archived using the Internet Archive because of their use of JavaScript or other technology. Others, like [[ https://web.archive.org/save/https://www.hyd.gov.hk/ | large parts of the Hong Kong government's website ]], actively or passively refuse connections from the Internet Archive's Save Page Now tool; others can't be archived because of their robots.txt; and still others, like the //New York Times//, deliberately show an error message even if the content has been loaded to prevent their paywall from being circumvented. Increasing the frequency of the Internet Archive's IRC bot (T199193) is not as helpful if it's still unable to archive a lot of pages. It would be really, really nice if the process of archiving links (regardless of the entity running it) could have fallback methods for archive attempts that don't work. (I know the Internet Archive does have a YouTube collection, but it's fairly difficult to actually get content into it, it seems to be only for copyleft/public domain videos, and I don't think the bot is capable of adding to it.) [[ https://en.wikipedia.org/wiki/Help:Archiving_a_source | w:en:Help:Archiving a source ]] lists five web archiving services, including the Internet Archive. All five of them have drawbacks which prevent them from being used to archive everything. * Archive.is has a limit of about 30 pages per minute, and actively tries to prevent mass archival. I know this because they've probably blocked my ISP because I tried to archive too many pages (don't worry about it; it only happened after tens of thousands of pages). However, it has several fairly good fallback mechanisms to archive pages, including the Google cache and some other websites. It's also permanently logged in to a Facebook account, and is the only one of the five to have a Tor hidden service. * Webcitation.org is fairly bad at archiving CSS, in my experience, and [[ https://www.webcitation.org/6zExGYP3x | also fails ]] on some Hong Kong government websites. It doesn't really seem to have any benefits over web.archive.org. * Perma.cc is only used in 62 English Wikipedia articles as of this writing. I don't know much about it but it has a limit of 10 pages per month per account. Perhaps @Green_Cardamom might know more. * Webrecorder is only used in 16 articles as of this writing. It's a nonprofit which was created this year, and as far as I'm aware is the only archiving service of the five which can save [[ https://webrecorder.io/wikipedia_1/youtube | YouTube videos ]]* (try loading the comments – you'll get to exactly where I stopped scrolling), [[ https://webrecorder.io/wikipedia_1/openstreetmap/20180716135655/https://www.openstreetmap.org/#map=14/51.5029/-0.1225 | OpenStreetMap's "slippy map" ]], and other interactive/dynamic content. It also has virtual machines, presumably for viewing outdated HTML. However, it would be more difficult to fix dead links with it because all archive links contain the name of the account which was used to create them (i.e. URLs can't automatically redirect like they do at web.archive.org). It would also probably be fairly difficult to crawl sites without support from the site owners. Having a ninety-something percent archival success rate is still not really ideal if it's out of millions of links per year, many of which will be gone within a few months of their addition to articles. The main reason I'm posting this here is that [[ https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(idea_lab)#Running_our_own_archive_service? | there is currently a Village pump (idea lab) ]] thread on the English Wikipedia proposing that the WMF runs their own archiving service. While I personally think this is somewhat unnecessary and would be an inappropriate use of WMF resources, especially when there are other non-profits which are already dedicated to web archiving, the current options are not satisfactory for archiving everything that should be archived. (* – the YouTube archival doesn't totally work because YouTube's links generate some sort of error and not all of the video quality settings can be chosen)