Page MenuHomePhabricator

Evaluation of link recommendations to orphan articles via link translation (Q3/Q4)
Closed, ResolvedPublic

Description

In https://phabricator.wikimedia.org/T293032 we developed a method to generate recommendations for new incoming links for orphan articles based on link translation. We showed that this can be used, in principle, to de-orphanize 4.9M out of the 8.4M orphan articles.
In this task we want to develop a procedure how to evaluate the recommendations.

  • Quantitative evaluation of link recommendations for incoming links to orphan articles against one or more baselines
  • (stretch) build a test-API to surface recommendations (at least for some languages) to allow for qualitative evaluation

Event Timeline

Update week 2022-01-10:

    • onboarding collaborator (Akhil) to the current state of the project and upcoming tasks
    • sketched a protocol for evaluating the link recommendations to orphans: dataset, task+metric, baselines
  • started to build a test-API to surface link recommendations for incoming links from link translation. in order to avoid scaling issues at this point, I am only yielding the top-3 choices and only for simplewiki.

Update week 2022-02-07:

  • starting to build the evaluation dataset of links that were added to orphan-articles in one month. considering to change data-processing pipeline from pagelinks-table (containing any link in the article) to html-dumps which allows to separate links from, e.g., navigation templates which are not good candidates for recommendation.

Update week 2022-02-14:

  • ongoing: building the dataset for evaluation of link recommendation to orphans. decided on an approach combining pagelinks-table with wikitext-dumps: while the latter contains only a subset of all links (see Mitrevski et al), it allows to keep track of which links appear in the text of an article (in contrast to links transcluded from templates which are less useful for recommendation).

Update week 2022-02-28:

  • solving some details when building the evaluation dataset about which links to include from the text for recommendation (e.g. remove links from infoboxes, include links from lists such as "See also" sections) and how to parse wikitext accordingly.

Update week 2022-03-07:

  • working to finish the dataset for the evaluation
  • collecting and starting to implement baseline models to demonstrate that task is difficult; for example, the orphan-template suggests to use the Find link-tool which is based on a text-search looking for direct matches to the title of the orphan article in other articles. while the tool works good in many cases, for orphan-articles there are often no resutsl (example).
MGerlach renamed this task from Evaluation of link recommendations to orphan articles via link translation (Q3) to Evaluation of link recommendations to orphan articles via link translation (Q3/Q4) .Apr 1 2022, 3:42 PM

Update week 2022-03-28:

  • evaluation of the link-recommendation is ongoing and will be completed during Q4
  • we have generated a ground-truth dataset of all links that were added to orphan articles in all Wikipedias comparing two consecutive snapshots
  • We are currently implementing the evaluation in terms of the recall@k:
    • given a true link (s_true,t) that was added to an orphan article as a target (t), our model generates a ranked list of the top-k links (s_1,t)...(s_k,t) to the orphan article as a target
    • averaging over all true links, we calculate how often the true link was among the top-k recommendations
  • our link-translation model will rank the recommended links according to the number of language editions the link already exists

Update week 2022-05-02:

  • finished first evaluation of link recommendation for orphan articles
  • evaluation data: 68,473 orphan articles that were de-orphanized between two consecutive snapshots (2022-01 vs 2022-02), we extract the in-links that were added to these articles as ground-truth. Note that this is less than 1% of the total number of orphan articles that exist (>8M).
  • we calculate recall@k, i.e. the fraction of times the added link was among the top-k suggestions from link-translation. Some example statistics for k=10:
wiki_dbrecalltotalrecall@10
enwiki45873890.0619840303153336
plwiki60923680.25717905405405406
arzwiki46422580.20549158547387067
viwiki154118230.8453099286889743
frwiki31018100.1712707182320442
  • next step: compare with (simple) baseline from, e.g., recommendations from standard graph-embedding

Update week 2022-05-09:

  • evaluating the baselines for comparison: i) simple heuristic based on reciprocating outgoing links; ii) nearest neighbors in embeddings of link network (currently training the embedding)

Update week 2022-05-16:

  • investigated different heuristic approaches for baseline comparison. I ended up with 3 heuristics we can use almost out-of-the-box
    • link-reciprocity: suggest any of the existing outgoing links as new incoming link (can be obtained easily from the pagelinks-table)
    • morelike: recommend similar articles from cirrussearch' morelike (the top-3 suggestions are shown in the mobile version as "related articles"). can be easily obtained by querying the MediaWiki API
    • findlink-tool: this tool suggests articles to link from based on text-search. it is listed in the orphan-template as a way to find suggestions how to de-orphanize articles.

Update week 2022-06-27:

  • figured out how to query morelike and findlink-recommendations as baselines
  • started summarizing and writing up the results on the evaluation of the link recommendation for the meta-page

Update week 2022-07-04:

  • running baseline evaluation
  • cleaning up repo and writing results for meta-page (ongoing)

Update weel 2022-08-08:

  • completed and compiled evaluation results of link recommendations for orphan articles (see spreadsheet)
  • will close task when tables are moved to meta (hopefully next week)

Added the results of the evaluation to the meta-page: https://meta.wikimedia.org/wiki/Research:Recommending_links_to_increase_visibility_of_articles#Results

  • High-level summary: The link-translation approach considerably outperforms all baselines and works particularly well in the case of smaller wikis and when surfacing only few recommendations.