Until resolved, you can add {{cbignore}} to keep the bot off the citation. That's what I did for the few I added. BTw ghostarchive.org uses the same Webrecorder technology on the backend. It won't work in every case as a substitute for Conifer, but many it will.

My workaround is compare the date of the diff (via MW API) with the date in the JSON and if they are too far apart assume the JSON is buggy data, ignore and log it. There is a massive log, now.

Thanks @bd808 - tool deletion request: https://phabricator.wikimedia.org/T308587

Hi, I would like to delete the tool entirely. If someone wants to have this service, it would best to start over with a new account and install.

This is a hard problem as the status can flip back and forth. As noted by Mark Graham above (Director Wayback Machine) archives that show excluded is like a curtain, the archive still exists in the Wayback Machine and could flip back to active in the future based on policy decision. The reason they are being added into wiki anyway is because IABot has a separate cache database and when it first detected that URL it was active. As a friend recently noted, one of the hardest things in computing is keeping accurate caches. The design of IABot is to use caching and not querying the WaybackMachine for every URL it encounters, which has pros and cons.

Testcases page: https://en.wikipedia.org/wiki/User:GreenC/testcases/ghostarchive

Block 12946111 https://en.wikipedia.org/wiki/Special:BlockList?wpTarget=%2312946111&blockType=&limit=50&wpFormIdentifier=blocklist is for User:Lallint .. confirming they have used the IABot tool before, last active 22:45 30 January 2022

Bot is converting short-form archive.today to Wayback links:
https://en.wikipedia.org/w/index.php?title=Elizabeth_Holmes&type=revision&diff=1065246449&oldid=1065173325

This is awesome. Confirming deployed AntiCompositeNumber's shadows.py to produce the list and the GreenC bot Job 10 ("shadows.awk") is back running, having just tagged 25 pages .

I don't remember what prompted creation of this ticket. I'll make a new ticket if seen again, with an example.

No idea. Feel free to adjust for the right audience I wasn't sure.

Domain now blacklisted - could also lift global live state to "none" and let IABot do dead link detection and fix normally but blacklisting will resolve faster.

I will process this domain for dead links, set them blacklisted (locks it in so global live state won't matter), then remove the global whitelist . Takes a while.

Fixed. https://ca.wikipedia.org/w/index.php?title=Cas_Peatge&type=revision&diff=27979994&oldid=27979974

it wouldn't need to be realtime up to date, so if the data is a bit stale, like updated weekly or monthly

@Amire80 that is a fascinating taxonomy you created. Glad you found this thread :) I'm having trouble understanding some of them as there are no examples. Intended space has an example, for example, so I understand where nowiki appears and might find those eg regex. []]{2}<nowiki/>[[]{2} - in fact there are 8 intended space on enwiki:

• https://en.wikipedia.org/w/index.php?search=insource%3A%2F%5B%5D%5D%7B2%7D%5C%3Cnowiki%5C%2F%5C%3E%5B%5B%5D%7B2%7D%2F&title=Special%3ASearch&profile=advanced&fulltext=1&ns0=1

If you ever decide to expand the description or a new column to include (more) examples where possible it would be very helpful towards developing tools and reports. It should be possible to detect many of them universally (like intended space) potentially making for a global bot/tool/report across all wikis.

Hi - I am also having trouble. In one day, it falsely reported over 2,000 edits on ukwiki as made by InternetArchiveBot - not a small number of false positives.

@Samwalton9 that's it. Thank you.

I see it was transcluded in from this edit to a template used on the page: https://ca.wikiquote.org/w/index.php?title=Plantilla%3ARingler&type=revision&diff=130519&oldid=98762

I believe it is an old bug long since fixed. If you see anything like it again it's not fixed, but the bot has edited cswiki a lot since then which is a good sign.

On zhwiki this turned out to be an IP filter.

This is now happening to me on zhwiki as of today.

Reporting same problem. Unable to add maintainers. This is urgent, unable to get a team of developers going on a project. Thanks.

I'm not really a JS person but suspect this might be solved by simply installing the software from scratch following the directions at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Node.js and making sure everything is in the correct directories. Ideally by someone familiar with JS and/or Node. Perhaps delete everything and start over with a fresh install. What I noticed is the current install it not in the right directories per linked directions.

Reposting bd808's comment: "Holding open an SQL select for hours (or even minutes) while you page through it for secondary lookups (the in-memory join as it were) is prone to a lot of interruption vectors." An SQL query that takes hours to complete, and is run daily, might run into aborts and lost data?

@Bstorm what happens when a new File: is uploaded to Commons that overlaps with a pre-existing File: on enwiki for example one uploaded years ago?

OK thinking it through.. for example monitor recentchanges on the Commons side (the larger) and compare against the full corpus on the Enwiki side. This is great because it reduces the size of Commons to a few entries. Thus if the file pre-exists on Enwiki and is then added to Commons it works. But, if the file pre-exists on Commons and is then added to Enwiki, rececentchanges on Commons would not see it. So there would have to be recentchanges for both enwiki and Commons in both directions. But at the same, you would also need the full corpus in both directions, leading back to the original problem of a very large list for Commons. Unless the idea is to build that list from scratch Day 1 save to disk and each day add to it from recentchanges. Or maybe I am missing something.

Shadows bot currently runs every 24hrs, so the AntiCompositeNumber SQL method takes under 6 hours is workable, in that sense. Slower than the 90 seconds join method. It seems like a lot, 6 hours a day, on a SQL query paging through, and more error prone given the time exposure and networking. I can confirm using an API method would take around 70 hours, the pywikibot method would probably be similar if API based. Estimating data size of 65 million page title is 1-3 gigabytes. It's a lot to request, retrieve and process daily even in the same datacenter.

Since learned it should be installed per: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Node.js
I have not had a moments time to try it, but suspect this convention would allow the proxy to see it.

After some various things, it now reports it is running

> translation-server@2.0.3 start /mnt/nfs/labstore-secondary-tools-project/translation-server/www
> node src/server.js

If anyone would like shell access to look around let me know.

The cronjob is throwing this error:

This might just be bad data in the db seems ok after correcting the db

It not longer does this (move the &dq text into the |quote field)

I've initiated an abandoned tool procedure so we can add additional maintainers and hopefully fix the tool. smith609 is reportedly busy IRL right now.

@ifried ok thanks my mistake had the wrong URL. Closing tix

The bot is patched. Closing ticket, if you still want to discuss reopen.

Checked the logs and found one other case at A. J. B. Johnston

It appears "title=none" is an undocumented bypass, otherwise gives a red error. It will skip on title=none.