In T276857#6896276, @Samwalton9 wrote:

A more fundamental learning might be that citations are needed at all. We could also consider alerting editors when they add new content but don't add a citation.

Great spot and agreed.

The research team developed a Citation Needed model that could do the heavy lifting on understanding whether a citation is needed for a given piece of text.

@Samwalton9 this is the first time I'm hearing of this...can you confirm this is the research you were referring to https://meta.wikimedia.org/wiki/Research:Identification_of_Unsourced_Statements ?

In T276857#6969784, @ppelberg wrote:

@Samwalton9 this is the first time I'm hearing of this...can you confirm this is the research you were referring to https://meta.wikimedia.org/wiki/Research:Identification_of_Unsourced_Statements ?

That's the one :)

Regarding the machine-readable storing of reliable/unreliable classifications, I have a couple of thoughts. First, @Newslinger has been working on a tool to take the English Wikipedia's table and turn it into a more usable format - looks like you can read more about that here.

Second, I've been feeling cautious about the idea of telling users explicitly what is or isn't reliable as they edit. On the one hand it seems like an obvious thing to do - the community already has this table with encoded community conventions which we could make available to new users more readily. On the other hand we would risk strengthening Wikipedia projects' biases around sourcing. We might want to make sure we design such a feature in a way that doesn't actively discourage adding sources which aren't in the list yet (i.e. don't train editors to look for a sign of approval that a source is definitely reliable). We already have a problem with editors not understanding what sources are reliable in different languages/countries/contexts (see an effort to alleviate this issue at Wikipedia:New page patrol source guide. Maybe this is an inherent tension with attempting to codify fuzzy norms and practices.

Thanks for the ping, @Samwalton9.

Here is an example of what the machine-readable data for the English Wikipedia's perennial sources list looks like in JSON form:

https://api.sourceror.org/v1/all_entries

The data is scraped and parsed from the perennial sources list. This format can be adapted for equivalent source lists on other Wikipedias. The Wikidata entry links to several non-English lists, two of which can be parsed to a machine-readable format:

• Persian Wikipedia: https://fa.wikipedia.org/wiki/%D9%88%DB%8C%DA%A9%DB%8C%E2%80%8C%D9%BE%D8%AF%DB%8C%D8%A7:%D9%85%D9%86%D8%A7%D8%A8%D8%B9_%D9%85%D8%B9%D8%AA%D8%A8%D8%B1/%D9%85%D9%86%D8%A7%D8%A8%D8%B9_%D9%85%DA%A9%D8%B1%D8%B1
• Chinese Wikipedia: https://zh.wikipedia.org/wiki/Wikipedia:%E5%8F%AF%E9%9D%A0%E6%9D%A5%E6%BA%90/%E5%B8%B8%E8%A7%81%E6%9C%89%E4%BA%89%E8%AE%AE%E6%9D%A5%E6%BA%90%E5%88%97%E8%A1%A8

On the English Wikipedia, the AbuseFilter and blocklist features are able to handle some use cases for this data (deprecation and blacklisting, respectively). However, there are limitations that reduce the effectiveness and hinder community acceptance of these technical measures:

• As of February 2021, the Wikipedia apps for Android and iOS are not able to display edit filters, according to the table in this noticeboard discussion.
• There is currently no way to apply blocklist entries to a selection of pages. All patterns on the blocklist apply to all pages on the project.
• Neither the AbuseFilter nor the blocklist provides a simple way to target or ignore content additions to particular sections of a page.
• As Samwalton9 mentioned, the messaging associated with these technical measures could be improved. The community can handle some of this, but it would be helpful to have more data available that could be incorporated in the template messages. For example, when a user adds a link that is either deprecated or blacklisted, the error or warning message should ideally show the paragraph surrounding the link.

Peter and I have been discussing using the Spam blacklist as a starting point for this, as entries there are more easily categorisable as obviously undesirable, whereas the perennial sources list has many entries with edge cases and nuances and is as-yet a few steps further removed from something the editor could easily parse (especially so because it doesn't exist on all Wikis). I did some investigation today looking through the English Wikipedia blacklist log and found that entries can be broadly categorised into the following (often overlapping) buckets: Spam, URL shorteners, and unreliable sources. The following is some notes on how ~200 randomly selected log entries broke down:

• Spam (40%): These were entries clearly designed to lead readers to some website selling a product, hosting a suspicious file, or otherwise of no encyclopedic value whatsoever.
• URL shorteners (35%): These are entries which introduced links to websites like bit.ly, youtu.be, or Google Amp. These are on the Spam Blacklist because they can disguise their destination, but I was surprised at the volume of hits this category receives. It's worth pointing out that many of these links might have been to spam sites, but I'm sure many were good faith edits.
• Unreliable sources (25%): These links appeared like they could be useful references for articles. While I'm sure many have been spammed or aren't even remotely reliable, I could imagine most of these link additions having been made in good faith by a new user.

I'm posting this here because I think this backs up the idea of the spam blacklist being a sensible place to start - if 90%+ of hits were clearly from spam bots I might have suggested another approach, but as much as 60% of spam blacklist hits are at least potentially being made in good faith and many more entries on the blacklist are to unreliable sources than I had previously thought.

We could imagine three lanes of guidance based on this categorisation, explaining that the user's edit won't successfully be saved, and then providing guidance to move away from a URL shortener, use a more reliable source, or to check that a link isn't complete garbage. This is prompting me to think about how we could facilitate categorisation on the Spam blacklist to match entries to these specific guidance paths; I'm not sure how that would work right now.

Related wish from this year's Wishlist: https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2023/Citations/Warn_when_adding_a_url_reference_that_matches_the_SpamBlacklist

It would be nice to have the "Perennial sources" list under the community config page so it can be turned off on the small and medium sized wikis, most of which do not have one.

The only global rule I am aware of is that on the blacklist are blog-sites, sites registered in whois with the same name as the article (especially websites of the company of an article) and of course the local and global spam blacklist.

@ppelberg as I saw that you have added the language-agnostic reference risk model card, please find the datasets with the risk scores for each domain in each wiki at https://analytics.wikimedia.org/published/wmf-ml-models/reference-quality/reference-risk

Per offline discussion with @MMiller_WMF: maybe a first step here could be evaluating the extent to which the content someone is attempting to Wikipedia is supported by what's written in the source they're citing.

Jotting down some notes from offline discussion with @Pablo and @Sucheta-Salgaonkar-WMF about the Language-agnostic Reference Risk model:

• The model is already available on LiftWing: https://api.wikimedia.org/wiki/Lift_Wing_API/Reference/Get_reference_risk_prediction#Anonymous_access.
• At present, the model accepts a revisionID and lang (e.g. https://en.wikipedia.org/w/index.php?title=Lux_(Rosal%C3%ADa_album)&oldid=1322859357) and outputs the following:
  • ps_label_local: the status that web domain is the perennial source list of that wiki (if exists),
  • ps_label_enwiki: the status that web domain in the English Wikipedia perennial source list (if exists),
  • survival_ratio : using data from the model_version (currently 2024-11), the survival ratio of that web domain when used as a reference on that wiki (i.e., proportion of the number of edits the domain stayed on the page over the total number of edits since addition). Values range from 0 to 1 (the closer to 0, the riskier),
  • page_count: using data from the model_version (currently 2024-11), in how many pages that web domain has been used a reference on that wiki,
  • editors_count: using data from the model_version (currently 2024-11), how many editors have used that web domain as a reference on that wiki.
• The model does not use ML but applies pre-computed scores of references in each wiki. Scores for the 2024-06 version can be found at https://analytics.wikimedia.org/published/wmf-ml-models/reference-quality/reference-risk
• The model is language-agnostic
• The model has not been tested with volunteers yet, the heuristic of using the survival of references comes from these ML experiments we ran https://arxiv.org/abs/2410.18803
• The model is using pre-computed scores for any single source (data folder, e.g., these are scores for frwiki). Therefore, Edit Check or any other service interested in scores for a single source could already use them directly as well.