InternetArchiveBot (https://meta.wikimedia.org/wiki/InternetArchiveBot) provides dead link fixing and reference enhancement services for over 100 Wikimedia wikis. As part of this work, it regularly inspects the wikitext source of articles to detect links, in-line references, and formatted citations. (In the future we will switch to Annotated HTML but for now we use wikitext).

InternetArchiveBot currently services a very large number of wikis, operating on a scale very few other bots in the Wikimedia world do, and our vision is to operate on every Wikimedia wiki. Short of becoming a Wikimedia production service this will require some amount of coordination between Wikimedia SRE and the bot operators at the Internet Archive. I am happy to manage the bot operator side of the relationship (though @Cyberpower678 should definitely be kept in the loop). I think with more proactive coordination there will be fewer surprises.

As part of this I would like for us to work and come to an agreement on what an appropriate level of concurrent requests would be. The idea is to ensure InternetArchiveBot can effectively serve its many users while preventing our operations from having a destabilizing effect on yours. From there we can figure out the action plan for implementing concurrency limits.