@Kerry_Raymond has independently raised this issue and provided some further examples in this thread.

I'm sharing an analysis on case-insensitivity on enwiki and simplewiki.

I use the recommendations that were rejected/accepted since June 1, 2025.

This analysis does not consider recommendations that users have skipped or have not yet seen.

• The root cause of case-insensitive recommendations is the anchors (link text in recommendations). We always use lower-case anchors in model training and inference. It comes with some advantages and disadvantages:
  • The advantage is that when an anchor is at the beginning of a sentence, it will have an upper-case first letter, although it refers to the same context, e.g., "Concert of the band" and "The band's concert."
  • The disadvantage is that case-sensitivity may reveal the context, e.g., the independent people, The Independent newspaper.

Before exploring possible next steps, I want to share some statistics:

• Among all accepted/rejected actions, the acceptance rate is 87%.
• Among all accepted/rejected actions where the link text matches the title only when case-insensitive, the acceptance rate is 83%. Case-insensitivity lowers the acceptance rate. However, we don't have data about the other way around.
• Among all accepted/rejected actions, 3% of the recommendations have a case-insensitive variant. Therefore, the scope is limited.
• In enwiki, 5% of the recommendations are not exact matches between the link and the target page, regardless of case sensitivity.
• Among all accepted/rejected actions where the link text matches the title only when case-insensitive, 33% of the recommendations have a case-insensitive variant.
• The Independent article is another good example. Among the recommendations for "The Independent" article, the acceptance rate of the link texts:
  • "the independent": 38%.
  • "The Independent": 50%.

Possible next steps:

• We can make the anchors case-sensitive and train/evaluate models. This experiment should give a general idea of how the models are improved and in which cases we decrease the quality of the recommendations. We should also reflect the changes in the inference. We can expect the anchors (8M) and bloom filters to increase in size.
• This issue also relates to a context problem. One of our features is the similarity between the pages. Using the context in a page could be too broad for some cases. We can lower the scope to sections or paragraphs.
  • We should use the same scopes.
    • Example: We use the first paragraph of the target article.
    • Then we can use the 'paragraph' where the text link occurs.
  • Using lower-level embeddings will increase the number of embeddings. We should review the average number of sections and paragraphs of articles.
• We may apply this change to a pre-defined set of wikis.

My short term suggestion is to make anchors case-sensitive and train/evaluate models. So that, we can analyse case where the performance increase/decrease.
Long term suggestion would be to have similarity between lower level embeddings (e.g. paragraph) as an additional feature.

Thanks, @OKarakaya-WMF!

My short term suggestion is to make anchors case-sensitive and train/evaluate models. So that, we can analyse case where the performance increase/decrease.

What level of effort would that be?
If it's limited work, then perhaps we can give this a try for the enwiki model and look at the precision and recall metrics and I can help gather community feedback?

Or if this sounds like significant effort, perhaps we should wait and only consider this type of change once we shift to the V2 model?

FYI @Trizek-WMF (since you asked about this today).

Thank you Kirsten. So far, I only heard about this at English Wikipedia.

hello @KStoller-WMF , and @Trizek-WMF ,

I think it should not be complicated to get offline scores for enwiki.
I'll get back to this and share the results soon after finishing some other tasks.
Please feel free to let me know if we should increase the priority.

@Chipmunkdavis has pointed to an additional instance of a partial name link in this edit.

thank you both @Sdkb and @Chipmunkdavis for reporting this issue,

I think making anchors case-sensitive and train/evaluate models will not fix this case.

Addalink first generates candidate links by creating ngrams (pair, triple, ... of words)

So both:
Agricultural Bank of France
Bank of France

are candidate links. As there is no page Agricultural Bank of France, Bank of France received some similarity with other pages and recommended.

Excited by this proposal, he settled in Molins de Rei and resigned from the Agricultural Bank of France by letter.

I think the best solution would be to generate candidates by using named entities rather than n-grams. (or use NER as additional filtering to ngrams)
So that Agricultural Bank of France will be an entity and we will make predictions only for Agricultural Bank of France.
I think it's easier to test NER as an additional filtering.

I'll include this to items to test/evaluate.

Thanks for that clarification, @OKarakaya-WMF! The partial name link issue (which the Agricultural Bank of France edit is an example of) is really the main capitalization issue we've been encountering — it's also what happened in the examples I cited above reported by Rich Farmbrough and Kerry Raymond.

Glad to hear that there is a potential solution. If there is a separate Phabricator task created for it, please add me.

@OKarakaya-WMF thanks soo much for the analysis you provided above re: opportunity sizing for case sensitivity! I'm curious if we have anything comparable for named entity recognition. I have two questions:
(1) Re: In enwiki, 5% of the recommendations are not exact matches between the link and the target page, regardless of case sensitivity. - I'm wondering how much of that 5% can be attributed to lack of named entity recognition.
(2) Do we have data to suggest that this might be an even more pervasive problem in non-English wikis, as (I believe) was the case for country and continent names?

(2) Do we have data to suggest that this might be an even more pervasive problem in non-English wikis, as (I believe) was the case for country and continent names?

We simply receive a lot more feedback from enwiki, but my guess is that this is an issue for all languages that include capitalization, while languages written in scripts without case distinction won't be impacted.

@IZapico-WMF - do you remember if this was an issue for eswiki in the past?

Note that this topic is again coming up on enwiki:
"If the capitalization in the suggested link is different from the article text (such as in your edit here), it's likely not a good link."

Note that a similar underlying issue is being surfaced on enwiki again.

@Sucheta-Salgaonkar-WMF - @Sdkb I discussed this task again as this comes up regularly as a complaint at English Wikipedia. I'm unclear how much effort a change like this would require... but I imagine this will become even more urgent as Suggestion Mode roles out and more people have access to link suggestions.

Perhaps we could consider work on this under DE1.1 next FY? @ppelberg

cool, thank you @KStoller-WMF ,

I'll check the examples again and share a proposed plan with estimation.

In T405185#11945641, @KStoller-WMF wrote:

@Sucheta-Salgaonkar-WMF - @Sdkb I discussed this task again as this comes up regularly as a complaint at English Wikipedia. I'm unclear how much effort a change like this would require... but I imagine this will become even more urgent as Suggestion Mode roles out and more people have access to link suggestions.

Perhaps we could consider work on this under DE1.1 next FY? @ppelberg

@KStoller-WMF: great spot. Let's talk more about this when we meet next/at the offsite?

In the meantime, I've added this ticket to the DE1 FY26/27 Objective Overview & Planning doc. Specifically, under the "Invest in technical infra to cope with scaling suggestions" heading within the DE1.1 [Peter] tab.

Just my 5 cents:

In T405185#11391268, @OKarakaya-WMF wrote:

thank you both @Sdkb and @Chipmunkdavis for reporting this issue,

I think making anchors case-sensitive and train/evaluate models will not fix this case.

Addalink first generates candidate links by creating ngrams (pair, triple, ... of words)

So both:
Agricultural Bank of France
Bank of France

are candidate links. As there is no page Agricultural Bank of France, Bank of France received some similarity with other pages and recommended.

Excited by this proposal, he settled in Molins de Rei and resigned from the Agricultural Bank of France by letter.

I think the best solution would be to generate candidates by using named entities rather than n-grams. (or use NER as additional filtering to ngrams)
So that Agricultural Bank of France will be an entity and we will make predictions only for Agricultural Bank of France.
I think it's easier to test NER as an additional filtering.

I'll include this to items to test/evaluate.

Maybe it would be good to teach the check to present links to other Wikipedias (using this template or this template) in such cases? Should I file a separate ticket for that?

Recently, Editing Suggestions started using the addalink model endpoint directly rather than fetching the suggestions from storage.
Therefore, we need to consider latency during the improvements.
On the other hand, the approach using the endpoint directly is temporary. So, they will switch using a storage in the next phases.

I share an example that could help us preventing "Bank of France" suggestion as it marks "Agricultural Bank of France" as an entity

https://huggingface.co/dslim/bert-base-NER?text=Excited+by+this+proposal%2C+he+settled+in+Molins+de+Rei+and+resigned+from+the+Agricultural+Bank+of+France+by+letter.

In overall, I think this takes:

• Testing a couple of NER models so that we ensure it does not harm or it improves offline scores.
• I think latency became important as addalink api is used in editing suggestions directly. Therefore, we need to assess:
  • Can we mimic the NER model with heuristics e.g. using capital letter or other hard-coded rules?
  • if the selected model is lightweight and it runs with a low latency without needing a GPU.
  • if they have large enough context to fit as much as possible. So that we can make less calls to NER model per article.
  • should we leverage existing hosted LLM models and use them as a NER model.

Therefore, my suggestion is to run a small experiment with followings:

• Mimic NER with heuristics e.g. using capital letter or other hard-coded rules. Noting that using capital letter is common in many languages and we can exclude some other languages.
• Try a small, large context, multi-language NER model.
• Report accuracy and latency scores and make a decision.

I think we can time box the experiment.
It time does not permit, we can try the first one (mimicing ner).

Amazing, thank you for investigating @OKarakaya-WMF!

run a small experiment

Is that something we can do now? I remember previously the concern was we didn't have a feasible way to A/B test suggestions from two different models.

If that's feasible in a lightweight way, let me know what you suggest! Does the Growth team need to set this up?

If it's not feasible in a lightweight way, I'm also open to keeping this as simple as populating a set of suggestions in a few languages for the current model and the new model and manually reviewing them with the support of Product Ambassadors and community members.

Hey @KStoller-WMF , Thank you very much for your comment!

Actually, I meant an offline evaluation by "running a small experiment" so that we can figure out which option helps more. I believe we will not need an A/B test for this problem as well.
I looked into what could be a quick solution with heuristics that we can review with Product Ambassadors and community members.

We get all candidate anchors in mention_original with their original text e.g. upper case ("naar de Aarde")
I think we can run a pre-filtering with:

• if the first and the last word in the ngram start with an upper-case. (So we include Bank of France).
• if the mention_original is in another other mention original in mentions, we will exclude it. For instance it should exclude "Bank of France" suggestion if "Agricultural Bank of France" is also in mention_originals set. We need to make sure there is no punctuation between the words and the larger mention is an anchor. So that we exclude ngrams such as "Germany and France" for instance.
• mentions are created per node so they don't have all possible ngrams in the article. This is good as we are interested in a narrow scope.
• We can discuss if we should check the location match. Probably it should work without a location match in many cases already.
• We can discuss if we should include/exclude some languages as some languages do not have uppercase letters.

@Urbanecm_WMF @Michael , What do you think? Please feel free to raise any comments.

Maybe it would be good to teach the check to present links to other Wikipedias (using this template or this template) in such cases? Should I file a separate ticket for that?

@Wellverywell, that's an interesting feature request! Since it's not AFAIU related to this issue, yes, I'd say file a separate task for it so it can receive dedicated discussion.

I've looked into the proposal in more detail:

• we need to consider that the anchors are lower-cased.
• linked_mentions is partially doing what we wanted to do. If Agricultural Bank of France was linked first, it would not link Bank of France as it's not possible :D I think we will need to do something similar including the prospect mentions but we need to be careful while defining the prospect mention and consider edge cases.

Have now come across an edit from today where "...as Ellen Katherine Bourke on..." was newcomer tasked into "as Ellen [[Katherine Bourke]] on...". No confounders about being at the start (or end?) of a sentence, no lower case "of" within the name. Needless to say, bad link. Given the wider issue is sliding into a year-long challenge, would it be viable to fix such simpler cases first?

Hey @Chipmunkdavis , thank you for reporting this.
can you share the revision id and the wiki id?
I'm planning to test some reported cases on the new model.

@OKarakaya-WMF Special:Diff/1366779612 on en.wiki