It is being done entirely for the benefit on Wikipedia, as explained above. Not because the site owner requested it. The site owner suggested this after I asked what they thought would be best, for the benefit of Wikipedia. The site owner could care less what we do, it does not matter to them. It's not a branding issue, if that is your concern.

To add, the owners of archive.today requested we at Wikipedia use archive.today as a safeguard against potential future outages of the other domains, such as archive.is, which have in the past been subject to problems.

This is even worse:

To be honest, this should be disallowed:

It's not cosmetic. Archive.today uses 5 or 6 domains such as .is .ph, etc.. these are where the content is located. The domain archive.today does not contain content, it's only function is to route requests to one of the other domains. Thus if archive.is becomes unavailable, the admins only need to reroute traffic to archive.ph .. however if the incoming link is archive.is then it won't function. Note that "becomes unavailable" is a domain-level outage, because this archive provider has had problems with domain registars and/or DNS resolvers that blackhole it for policy reasons.

A domain is dead. The bot didn't do anything.

We still have over 1.2 million instances of webcitation.org .. half are on two sites ruwiki (441k) and ukwiki (216k). Other large sites include Chechnya (64k) and Armenia (50k). It's a bigger problem for them. Enwiki is mostly converted, only 37k left, there used to be many 100s of thousands. Every month the totals drop a little (all wikis last month -27k). In terms of enwiki and reFill I would skip processing when encountered.

Hi, my tools here are awk and tcsh scripts mostly, that further invoke other unix tools like sort, uniq etc.. that also depend on a library I wrote called BotWikiAwk which has environment variable and other awk scripts.

If you need a testcases page let me know.

There are 14 links on Enwiki and Jpwiki has over 1,000

Per my duplicate task.. it's might be too much for the bot to check every http request two times.. but if we know an entire domain has moved to https, then it should be possible to include an optional feature in the interface to tell the bot it should check a URL for https if it's getting a 404 on http. If it gets a 200 on the https, then change the source URL to https and don't add an archive. It could go further by checking existing citations with url-status=dead and converting to live, but this is probably beyond the bot's scope right now.

It's unclear if IABot is still adding to discussion pages. I see in the above example the last one was dated December 2022. Current edits by the bot are not adding to discussion pages. This on Italian wiki. On enwiki is stopped many years ago.

Bots are not really "adding" blacklisted links, the links were already there. We can continue to block bots from doing their work, but the consequences IMO are much more severe than the edge scenario outlined above.

Until resolved, you can add {{cbignore}} to keep the bot off the citation. That's what I did for the few I added. BTw ghostarchive.org uses the same Webrecorder technology on the backend. It won't work in every case as a substitute for Conifer, but many it will.

My workaround is compare the date of the diff (via MW API) with the date in the JSON and if they are too far apart assume the JSON is buggy data, ignore and log it. There is a massive log, now.

Thanks @bd808 - tool deletion request: https://phabricator.wikimedia.org/T308587

Hi, I would like to delete the tool entirely. If someone wants to have this service, it would best to start over with a new account and install.

This is a hard problem as the status can flip back and forth. As noted by Mark Graham above (Director Wayback Machine) archives that show excluded is like a curtain, the archive still exists in the Wayback Machine and could flip back to active in the future based on policy decision. The reason they are being added into wiki anyway is because IABot has a separate cache database and when it first detected that URL it was active. As a friend recently noted, one of the hardest things in computing is keeping accurate caches. The design of IABot is to use caching and not querying the WaybackMachine for every URL it encounters, which has pros and cons.

Testcases page: https://en.wikipedia.org/wiki/User:GreenC/testcases/ghostarchive

Block 12946111 https://en.wikipedia.org/wiki/Special:BlockList?wpTarget=%2312946111&blockType=&limit=50&wpFormIdentifier=blocklist is for User:Lallint .. confirming they have used the IABot tool before, last active 22:45 30 January 2022

Bot is converting short-form archive.today to Wayback links:
https://en.wikipedia.org/w/index.php?title=Elizabeth_Holmes&type=revision&diff=1065246449&oldid=1065173325

This is awesome. Confirming deployed AntiCompositeNumber's shadows.py to produce the list and the GreenC bot Job 10 ("shadows.awk") is back running, having just tagged 25 pages .

I don't remember what prompted creation of this ticket. I'll make a new ticket if seen again, with an example.

No idea. Feel free to adjust for the right audience I wasn't sure.

Domain now blacklisted - could also lift global live state to "none" and let IABot do dead link detection and fix normally but blacklisting will resolve faster.

I will process this domain for dead links, set them blacklisted (locks it in so global live state won't matter), then remove the global whitelist . Takes a while.

Fixed. https://ca.wikipedia.org/w/index.php?title=Cas_Peatge&type=revision&diff=27979994&oldid=27979974

it wouldn't need to be realtime up to date, so if the data is a bit stale, like updated weekly or monthly

@Amire80 that is a fascinating taxonomy you created. Glad you found this thread :) I'm having trouble understanding some of them as there are no examples. Intended space has an example, for example, so I understand where nowiki appears and might find those eg regex. []]{2}<nowiki/>[[]{2} - in fact there are 8 intended space on enwiki:

• https://en.wikipedia.org/w/index.php?search=insource%3A%2F%5B%5D%5D%7B2%7D%5C%3Cnowiki%5C%2F%5C%3E%5B%5B%5D%7B2%7D%2F&title=Special%3ASearch&profile=advanced&fulltext=1&ns0=1

If you ever decide to expand the description or a new column to include (more) examples where possible it would be very helpful towards developing tools and reports. It should be possible to detect many of them universally (like intended space) potentially making for a global bot/tool/report across all wikis.

Hi - I am also having trouble. In one day, it falsely reported over 2,000 edits on ukwiki as made by InternetArchiveBot - not a small number of false positives.

@Samwalton9 that's it. Thank you.

I see it was transcluded in from this edit to a template used on the page: https://ca.wikiquote.org/w/index.php?title=Plantilla%3ARingler&type=revision&diff=130519&oldid=98762

I believe it is an old bug long since fixed. If you see anything like it again it's not fixed, but the bot has edited cswiki a lot since then which is a good sign.

On zhwiki this turned out to be an IP filter.

This is now happening to me on zhwiki as of today.

Reporting same problem. Unable to add maintainers. This is urgent, unable to get a team of developers going on a project. Thanks.