Hey,

I think one solution could be adding a pre-calculating incoming_link_count of a page to the dataset. So that we can use it for filtering.

• Do not suggest a link to a page if it already has more than N links.
• Do not suggest a link to a page if the number of links to the page is in X percentile.

We can pre-calculate N and X as well and if needed we can make it configurable.

Hmm, @OKarakaya-WMF, would there be any way to test out how such a filter would affect the suggestions?

My instinct is that that approach could be challenging, as the pages that need to be avoided to prevent overlinking are the pages on the most generic topics, not the pages with the most incoming links, and those datasets do not overlap all that well. Here is a (dated) list of the pages with the most incoming links on en-wiki — some (such as "United States") are definitely generic, but others have lots of links for idiosyncratic reasons (e.g. "IUCN Red List" appears because it's linked from the animal infobox as part of the line on a species' conservation status) and would be fully appropriate links to suggest (which is indeed why they've been used so much).

One alternative might be to use the Vital Articles list, which is closer to (although still not precisely) a list of the most generic topics. I'd say that any non-biographical article at level 3 or above would be pretty safe to exclude as an overlink. The VA list is maintained by en-wiki, although it's not intended to have any English-language focus; the list of articles every Wikipedia should have is at Meta but is less well-maintained.