@OKarakaya-WMF Curious if you have any thoughts on this topic!

Of course, this would be more useful if the model itself was retrained regularly (and reingested every now and then). T429417 is a semi-related task to this.

Hey @Urbanecm_WMF , I think this is a good idea!

T415623 is also related as this approach should expand the training dataset beyond the first sentence.
It should not introduce much noise as long as accepted/rejected suggestions have a common logic (or may need a kind of clean up that we can figure out).
Noting that one edit may have multiple suggestions.
I think we keep last 3 months in event.mediawiki_structured_task_article_link_suggestion_interaction

I think it's definitely worth to try in an offline dataset.

I'm currently working on a related analysis (T435526) I'll add your suggestion to the final report and share.

Current model depends on 6 features.
If accepted/rejected suggestions have a common pattern that we can explain with the existing features, it should improve the results.
Otherwise, we may need new features to represent the new relations.
Therefore, I think it'd be good look first if we can find some common patterns in the accepted/rejected/reverted actions.

Reverted edits from "newcomer task add link" are 1% of all addalink onboarding tasks edits. So rejected links (20%) may give more insights.

Reverted edits from "newcomer task add link" are 1% of all addalink onboarding tasks edits. So rejected links (20%) may give more insights.

We have a lot more data on "rejected links", but I would hypothesize that it's not as accurate as the reverted edits data. The rejected links might include good links too, as rejections are likely made my newcomers that are more likely to make mistakes. While the reverts are made by experienced Wikipedians that are unlikely to revert a "good" newcomer edit.

I just wanted to note that we received feedback from a trusted user on enwiki supprting this task:

is it at all possible to retain lists of what links were rejected or reverted? One would hope that could be used in training data at some point to improve the task success rate.

Hey, I've added an implementation section to the task.
I can do it after creating the new features.

Meanwhile, I'm curious if we have another task not to suggest previously rejected/reverted links.
I think training with the user decision data may not replace a hard filter.

In T435831#12296680, @OKarakaya-WMF wrote:

Meanwhile, I'm curious if we have another task not to suggest previously rejected/reverted links.

We are already filtering for that on GrowthExperiments side. T429417: Add Link should not make a recommendation if one has already been accepted or declined (Limit an article to only one Add a Link suggestion) exists to enlarge that filtering to not suggest any article again (basically to prevent the model from slowly learning from itself, eventually oversuggesting). Does this help?

Thank you @Urbanecm_WMF , looks great and clear!

We should note a couple of points about this task:

• We don't have suggestion to edit mapping (suggestion_id to revision_id) in hive tables (at least I could not find it). Previously we have tried to match it with (date + user + page). I don't know if we have it in Growth team's tables.
• One edit may have multiple accepted suggestions. So when an edit is reverted, all suggestions in the edit are reverted. Therefore we don't actually know which suggestion(s) in the edit was wrong and we assume all of them were incorrect.

What do you think @Urbanecm_WMF ?

In T435831#12307133, @OKarakaya-WMF wrote:

We should note a couple of points about this task:

• We don't have suggestion to edit mapping (suggestion_id to revision_id) in hive tables (at least I could not find it). Previously we have tried to match it with (date + user + page). I don't know if we have it in Growth team's tables.

We have this data in growthexperiments_link_submissions:

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select * from growthexperiments_link_submissions where gels_feedback = 'a' order by gels_revision desc limit 1\G
*************************** 1. row ***************************
     gels_revision: 1373575771
gels_anchor_offset: 15876
       gels_target: 2393053
         gels_page: 10975930
gels_edit_revision: 1373979713
         gels_user: 55360152
gels_anchor_length: 17
     gels_feedback: a
1 row in set (0.001 sec)

Rejections don't have the gels_edit_revision, but that's because such decisions don't result in a revision at all. Does this help?

• One edit may have multiple accepted suggestions. So when an edit is reverted, all suggestions in the edit are reverted. Therefore we don't actually know which suggestion(s) in the edit was wrong and we assume all of them were incorrect.

Yeah, I'm afraid there's no better interpretation, unfortunately :(. To make things worse, if someone does the right thing (unlinks only the wrong recommendations), the mechanisms for detecting reverts don't notice that at all, as that isn't a revert to begin with. Personally, I think treating reverts as a negative sign for all recommendations is a reasonable approximation, but if we can take a look at "did the link disappear" (regardless of whether in a revert or naturally), that would be even better. Hope this makes sense.

One edit may have multiple accepted suggestions. So when an edit is reverted, all suggestions in the edit are reverted. Therefore we don't actually know which suggestion(s) in the edit was wrong and we assume all of them were incorrect.

Not sure it's helpful, but FYI dewiki, and more recently enwiki, limited the "Add a Link" task to only ONE link suggestion per task. Meaning recent reverts on enwiki and dewiki should match just one specific link suggestion.

This "Maximum number of links to show per task" setting is community configurable, so there might be other wikis with this setting:

I've collected user decisions (accept/reject/revert):
https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/blob/user_decisions/training/add_a_link/src/add_a_link/notebooks/analysis/build_user_decision_dataset.ipynb?ref_type=heads

Next step is to convert them into the training set format and analyse on the model features.

keeping the progress in a draft merge request:

https://gitlab.wikimedia.org/repos/machine-learning/ml-pipelines/-/merge_requests/94

we have collected 1 year of data from simplewiki.
We set labels as:

• accepted: positive.
• rejected and reverted: negative.

{F105270579, width=400}

Afterwards, we have generated features for the suggestions.

w2v, context_w2v and link_probability have the highest correlation with the labels.
Compared to the other features such as ambig, ngram, the features above seem be better to split positive and negative labels.

https://phabricator.wikimedia.org/T437264#12297442

As a next step:

• get same scores for enwiki.
• try a couple portions for the training+test dataset e.g. 20% user decisions , 80% training etc.
• try 2 labels and 3 labels.
• Final decision on how to use the user decisions dataset and the model.

Dataset for enwiki:

We have the highest correlation from link_probability, context_w2v and w2v respectively.

The graphs below show that the new features can help to separate positive and the negative samples

I'll add my notes soon!

enwiki: