Analysis

We have received several feedback from community and run an analysis based on the community feedback in scope of the epic T434350
We analyse "newcomer task add link" edits (create/revert edit) in last one year, and the user actions (accept/reject suggestions) in last 3 months.

Some terminology we use in this post for clarity:
target: target article.
text: text in the article.
edits: user edits such as create and revert.
actions: user actions in the onboarding tasks such as accepting and rejecting suggestions.

Overall results

We have 80.8% acceptance rate and 1.7% revert rate.

{F100195391, width=700}

Monthly revert rates remains stable.

Case1 - Case-insensitive match on a capitalized multi-word target (without a model change)

We looked into several features of the reverted suggestions (capitalization,
probability, location in the article, revert comments, etc.) to see whether
reverted add-a-link edits share a structure. One pattern stands out:

• case-insensitive exact match, and
• target is multiple words, and
• every word in the target starts with a capital letter, and
• at least one word in the text does not start with a capital letter.

Example edits:

• https://en.wikipedia.org/w/index.php?&diff=prev&oldid=1360300286 ("so much fun")
• https://en.wikipedia.org/w/index.php?&diff=prev&oldid=1359936045 ("an evening visit")

Accepted is 80.8% of all suggestions, rejected 19.1%, and reverted 1.7% of published
add-a-link edits. Normalizing to share of all suggestions:

• Accepted (cost): 3.5% = 3,246 suggestions
• Rejected (gain): 16.6% = 3,677 suggestions
• Reverted (gain): 9.2% = 133 suggestions

Still worth doing as a quick win via T434259. Noting that it will fix similar cases to "listen to the music" but not the cases similar to the "secondary school".
We will still need NER ( T405185 ) for the similar cases.

• Accepted suggestions:

{F100035308, width=400}

• Rejected suggestions:

{F100035388, width=400}

• Reverted suggestions:

{F100035449, width=400}

In overall it's good to keep prediction probability high. However, there is no big gap in prediction probability among accepted (0.76), rejected (0.74), reverted (0.74) edits.
This is one of the indicators to improve the model.

Edit comments

We have looked into some tags and keywords in the reverted edit comments.

{F100035799, width=400}

Please see the abbreviations in the links below:
https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_abbreviations
https://en.wikipedia.org/wiki/Wikipedia:Banning_policy#Edits_by_and_on_behalf_of_banned_and_blocked_editors

Overlinking is mostly related to our investigation.

We have also search for keywords in the revert comments:

• overlink

Overlink appears in the reverted comments by 5%.

• Incorrect ("correct", "wrong", "relevant", "bad link")

Incorrect appears in the reverted comments by 14%. On the other hand, an edit may be incorrect due to several reasons e.g. partial editing, wrong person linked, incorrect place name etc.
We will need to investigate more by using edit diffs in addition to the article, link, text, target by using a semantic way to categorize the reverted edits further.

Overlinking

Overlinking is another recurring theme in reverted edits, though it is hard to
measure directly. At least 5% of revert comments explicitly mention "overlink"
(see Edit comments above), but most overlinking reverts do not use the word, so
5% is a lower bound; qualitative review of the reverted set suggests it is
more common than the comments alone show.

From the guide: https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Linking#What_generally_should_not_be_linked

*A good question to ask yourself is whether reading the article you're about to link to would help someone understand the article you are linking from. Unless particularly relevant to the context in the article, words and terms understood by most readers in context are usually not linked. Be conscious of your own demographic biases when determining whether certain terms have this level of recognizability – what is well known in your age group, line of work, or country may be less so for others.*

Some article types are already excluded (e.g. country, units of measurement).
The open question, from MOS:LINKING, is how to identify links that are well
known and would not help a reader understand the article.

Two data sources can inform this, with a trade-off:

• Reverted edits are rare (~1.7% of published edits) but carry a reason in the

revert comment.

• Rejected suggestions are more (~20% of all suggestions) but reasonless,

since we do not ask why a suggestion was rejected.

Candidate directions:

• New features: targeting common-knowledge links (see Feature Engineering

below). If a feature cleanly separates overlinking cases, we could apply it as
either a hard filter or a model feature.

• Re-sampling: the training data across accepted / rejected / reverted

actions, as suggested by @Urbanecm_WMF. This helps the model learn user
behaviour more broadly, but the gain may be limited to existing features, so it
is best combined with new features rather than done alone.

• Banlist: extending the existing article-type exclusions, either

automatically by using article properties or via a
human-curated list.

I think we can discuss the direction with @Sdkb and decide. I will share the
reverted edits list as a CSV alongside this analysis.

Overlinking examples:

• https://en.wikipedia.org/w/index.php?diff=prev&oldid=1363525849 Empty set: likely a well-known term
• https://en.wikipedia.org/w/index.php?diff=prev&oldid=1362173157 Baghdad: major city
• https://en.wikipedia.org/w/index.php?diff=prev&oldid=1361982836 well-known terms/region: Human behavior, Job performance, Northeastern United States
• https://en.wikipedia.org/w/index.php?diff=prev&oldid=1363587682 borderline, looks acceptable: Pathos, Lyric poetry, Sonnet
• https://en.wikipedia.org/w/index.php?diff=prev&oldid=1367004140 well-known terms: Power set, If and only if, Uncountable set

Feature Engineering

Current features:

• freq: How many times was the link used with this text
• w2v: W2V Distance between the source and target page
• ambig: How many different links where are with this text
• kur: Skew of usage text/link distribution
• leven: levenshtein distance between the text and the link.
• ngram: number of words in the text.

Suggested features:

• number of times the text appears unlinked: common terms such as "education"

should appear often without a link, which may flag common-knowledge words.

• number of times the target article is viewed: common-knowledge articles

should have higher pageviews. Joinable from pageview data, at some cost.

• article's category: common-knowledge articles may be assessed in their categories

e.g. In terms of link count and view count, is Empty Set a well known page in math category, is Baghdad a well-known city in location category.

• context vector: we currently use article-level embeddings. Capturing the

context around the suggested link should give more focused suggestions. Two
options:
Paragraph/sentence-level embeddings, to judge whether the text relates to
the target article. Costly, since it needs an embedding per paragraph.
Nearby links, reusing the existing embeddings on the assumption that nearby
links are more related to each other. Steps: take the N nearest links to the
text, fetch their articles, embed them, and average the N embeddings.

The context vector may also help with incorrect person suggestions (a fictional
character matched to a real person), also reported in T409138:

• https://en.wikipedia.org/w/index.php?&diff=prev&oldid=1359977232 Graham Sutherland (probability 0.74)
• https://en.wikipedia.org/w/index.php?&diff=prev&oldid=1363168500 Dr. John
• https://en.wikipedia.org/w/index.php?&diff=prev&oldid=1365655317 Big Tom (probability 0.68)

This issue is also reported in T409138 (two Jane Smiths who are writer and swimmer)

NER

NER reported in T405185 can solve issues related to partial linking.
We will need to categorize the edits by using edit diff in order to understand how common this problem is.
I share an example from the reverted edits below;

Public Service Association
He is not a full-time author; he worked as a senior Trade Union official, with his role until mid June 2015 being the Chief Industrial Officer of the Public Service Association of South Australia.
https://en.wikipedia.org/wiki/Public_Service_Association_of_NSW
https://en.wikipedia.org/w/index.php?&diff=prev&oldid=1357663941

{F100043506, width=500}

Conclusion

The add-a-link onboarding tasks (newcomer task add link) has an ~80.8% acceptance rate and a stable ~1.7% revert rate. All figures are enwiki only.
Suggested next steps are:

• Step 1: Case sensitivity (T434259). A quick win, but modest.

We can discuss which languages should have this fix. We currently have enwiki in scope.

• Step 2: Feature engineering and dataset re-sampling. Add the suggested features

(common-knowledge / overlinking signals, context vector) and re-sample training
data across accepted, rejected, and reverted actions in addition to the existing logic.

addalink reverted edits in last 3 months:
https://docs.google.com/spreadsheets/d/1PBMXquBip4eGbcENgPtINseNXIdHV1fVZPPCXedBnJw/edit?gid=1404651149#gid=1404651149

ozge opened https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/92

feat: addalink - add reverted/not-reverted suggestion analysis

pull request for review: https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/92

@kevinbazira

In T435526#12260892, @OKarakaya-WMF wrote:

pull request for review: https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/92

@kevinbazira

Thank you for working on this @OKarakaya-WMF.

CI is failing: https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/pipelines/221762
but it's not because of the changes in your MR.

It's failing because it couldn't reach mirrors.wikimedia.org similar to the issue we solved in T429675.

IIUC this can be solved by bumping the workflow_utils ref under training/add_a_link/.gitlab-ci.yml from v0.25.0 to a newer release whose conda.yml uses a new base image that doesn't rely on the decommissioned mirrors.wikimedia.org.

thank you @kevinbazira , CI is fixed: https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/92

In T435526#12279799, @OKarakaya-WMF wrote:

thank you @kevinbazira , CI is fixed: https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/92

Thank you for fixing CI, @OKarakaya-WMF. The merge request LGTM, approved!

ozge merged https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/92

feat: addalink - add reverted/not-reverted suggestion analysis