See a report for the Demographics of India here. Checked a few "Not supported" manually, a few are due to broken links (e.g., in reference [5] this link probably used to point to mortality rate data but now it doesn't), some are probably due to editors updating figures without adding references (ref [16], 2021 CIA worldbook supporting 2024 median age numbers), others are false positives.

Not the wrong place, this is why this ticket exists.

Answering the open questions

Thanks for looking into it. My Wikimedia developer account username is also Alaexis.

Quick question re "Edit: When tapped, your cursor will be placed/focused at the start of the claim"

1. As T425808 demonstrates, there are many instances where a citation might contain multiple URLs and for those URLs to be embedded within templates. How will this Suggestion decide which URL(s) to evaluate?

Thanks @Chlod !

@Chlod, I came here from https://phabricator.wikimedia.org/T399642, what you've done is very impressive! We've been building a source verification tool and we'd love to access TWL sources in the same way the Copyvio Detector accesses it - but rather than checking for copyright violations we're chekcing whether sources support claims. Currently the tool only works with online sources and we've been getting a lot of feedback "can you make it work with TWL?"

@SEgt-WMF thanks for sharing this, both initiatives are super relevant to the source verification tool we've been building. Will reach out directly to the contacts

While now it needs an external LLM, there are plans to host LLMs internally T418350, this could be a great use case.

Btw it's def a good idea to add the article title and paragraph text to the context. One of the failure modes I encounter is when a claim "In 2026, she appeared on the game show Countdown in dictionary corner.[12]" is classified as "Not supported" because the model has no idea who "she" is.

Thanks for sharing the updates. Looking forward to the results!

Deploy the model-server in the LiftWing production namespace to provide an internal production endpoint for wider use.

Re the tensions between performance and nuance

Thanks!

@sbassett , could you whitelist the Anthropic, OpenAI and PublicAI endpoints? There are more scripts that stooped working, see https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_AI_Tools#c-Phlsph7-20260308103900-AI_scripts_not_working_anymore_following_CSP_changes

Great news! Running it like Suggestion Mode would be amazing. Is it something that WMF infrastructure can support now? This would really move the needle. The reach of user scripts is limited (mine has 14 active users).

api.publicai.co (see the documentation here https://platform.publicai.co/docs)

specifically api.anthropic.com and api.openai.com

Thanks for the quick turnaround. I'm happy to confirm that the tool works now... partially. I forgot that I also made calls to Anthropic and OpenAI. Since @Polygnotus also mentioned them, I think it would be good to whitelist them too.

I think it's quite likely that many scripts have been affected. I understand that Wikipedia:WikiShield (an anti-vandalism tool with 70 active users) also no longer works, @LuniZunie - please confirm

I'd suggest renaming this task to something like "Support user scripts accessing external data". the current title frames whitelisting as *the* solution, while the right approach is still being discussed.

@Polygnotus thanks for you comment. On the second though I think that you're right and toolforge is not a good solution.

I think that hosting on toolforge could be a good medium-term solution as many other applications need to fetch website contents or access open-souece LLMs

I've fixed the logging so now we'll start collecting data.

@Isaac thanks for sharing the statistics and impressions. Good to get a confirmation that this is solving a real problem for editors reviewing recent changes or articles for creation. It's tempting to imagine an edit filter but it needs to be considered carefully - there will be false positives and errors and we don;t want to discourage new editors. Another modality is a bot that flags citations that fail verification by adding tags or by posting at the talk pages.

Thanks for the detailed response

@Isaac , the short answer to your second question is that everything that I couldn't classify as "Supported" or "Not Supported" ended up as "Partially supported"

Re 1. it's certainly feasible technically. I thought about it but didn't do it to avoid having to think about privacy. I only see the number of requests to my proxy, it's about 20-30 per day and I'm not entirely sure they are all real. I'd rather not make it opt-in since most people won't click on it, and given the total volumes we'll hardly get any data. Do you think that collecting data by default would be okay if I don't store any PII-adjacent stuff (no user names, only verification results) and add a disclaimer to the UI? Or maybe you have better ideas how to solve this.

Thanks for sharing the results, very interesting!

I've tested a few models and written up the results here: https://github.com/alex-o-748/citation-checker-script/blob/main/Citation%20Verification%20-%20LLM%20Benchmarking.md

I've been using AI source verification (one of the tools mentioned above) and also built a standalone version for non-wiki editors https://wiki-cite-checker.replit.app/

In T336905#11452620, @Sj wrote:

And we are making Apertus and Olmo-Think and other fully public models available via API for free (though with rate limits to prevent saturation) vua the Public AI Inference Facility https://chat.publicai.co

Speaking of use cases, there are quite a few tools here https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Tools#Projects. For now they require API keys and hence their adoption is very limited. If some kind of LLM credits were available to established editors that would've been awesome!

thanks!