Page MenuHomePhabricator

Add a link: evaluate link recommendation (Feb 14 2020)
Closed, ResolvedPublic

Description

In T241395, the Growth team ambassadors evaluated a set of articles that had recommendations for blue links added by an algorithm developed by @DED. Since then, improvements have been made to the algorithm, and we are ready for the second round of evaluation.

This time, instead of using a Google Doc, the original articles were loaded to Test Wikipedia, and then the links were added in a revision. Therefore, the wiki diff interface can be used to see which links were added by the algorithm. Though this may make it easy to see the diff, we're not yet sure the best way for the ambassadors to indicate whether the link recommendation was good or bad. One idea is for ambassadors to add these templates next to the links that came from the algorithm:

{{Done}}: next to links that are good ones.
{{Not done}}: next to links that should not be made.

Here's an example article where those templates are used to do this: https://test.wikipedia.org/wiki/Test_Planet

When doing this, please note that not all the links in the article come from the algorithm; some were already there before. This can be seen from the latest diff. Note that all the links will be red (not blue), because the corresponding articles were not created in Test Wikipedia.

Below is the list of articles for each language.

Czech

Overall: 145/168 (86% good suggestions)

Arabic

Overall: 129/151 (85% good suggestions)

Vietnamese

Overall: 68/132 (52% good suggestions)

Korean

Overall: 59/76 (78% good suggestions)

English

Overall: 56/84 (67% good suggestions)

Details

Due Date
Feb 19 2020, 8:00 AM

Event Timeline

@revi @PPham @Dyolf77_WMF @Urbanecm -- this task is ready for you. I know this may be confusing, so please let me know if you have any questions.

We would like to have this done by Wednesday, Feb 19.

@Urbanecm -- you are usually good at coming up with a good system to do this sort of work. If you think of a good idea for how to record the evaluations, please let us know.

Vietnamese is finished.

However I'd love to be able to add comments like in the first round. Some of them are in between, not really bad but not good either. For example, in this article, "... tái hợp trong chất bán dẫn", the word "bán dẫn" is linked to the article "chất bán dẫn", so the algorithm leaves out the word "chất" (in italic) right in front of the linked word? The target article has 3 words and the algorithm only picks 2 although all 3 words appear? But it would be too strict to fail it, because linking "bán dẫn" only is still acceptable, so I still marked it "done".

The algorithm looks better (or is it because there're more articles to assess now?), but it still picks random and general words and breaks words into pieces (just like in the example above).

@PPham Thanks for the feedback! I am separating words that are formatted or in quotes because I took it that the added style is meant to highlight (or single out) the word. This is more of a rule that I apply to all languages.
More articles will come. Thanks again.

@Urbanecm -- you are usually good at coming up with a good system to do this sort of work. If you think of a good idea for how to record the evaluations, please let us know.

Adding {{done}}/{{not done}} isn't a bad idea. @PPham and whoever else wants to can add comments using <!-- bla bla bla --> (@DED would have to look for the comments in source, but since the articles usually don't contain any comments, it shouldn't be a problem.

(for my own convinience, I've created {{maybe}})

Urbanecm updated the task description. (Show Details)

First two articles done, I'll continue tomorrow. To be honest, I'd appreciate if future tests of the algorithm can be done through Google Docs, where I can provide any additional information I believe that can be useful.

@PPham Thanks for the feedback! I am separating words that are formatted or in quotes because I took it that the added style is meant to highlight (or single out) the word. This is more of a rule that I apply to all languages.
More articles will come. Thanks again.

Hi! Okay but as you can see the cases in my language are not like that at all. Nothing is styled differently from the word before or after it, yet it's still left out.

Please check this article: in this case you may argue that the full word ("Đại học Koln" = Koln University") isn't created yet in my language so the algorithm just pick the half that exists ("Koln"), but what about in this case? The algorithm only links "bệnh ung thư" (cancer) and leaves out "vú" (breast) while obviously we have the article "bệnh ung thư vú" (breast cancer). Same goes with this case (links only "bán dẫn" and leaves out "chất" while actually "bán dẫn" is a redirect to "chất bán dẫn").

Well I don't know about other language but Vietnamese is the kind of language in which some words need more than 1 "word" to have a meaning ("University" = "Đại" + "học"), you can't break them in 2 as they will no longer make sense. On the other hand, in English each word has it own meaning already, so... Ugh, it's hard to explain this but I hope it helps somehow.

@PPham Yes, I understand, this is extremely helpful, thank you so much. I will incorporate this observation.

@PPham @Urbanecm @Dyolf77_WMF @revi -- thank you for the work you've done so far on this. I have two updates:

  • @DED generated some more pages so that each languages has 20 pages. This will give us enough information to evaluate with. I'm sorry to add more work for those of you who have finished, but I hope you can also go through the articles I added to the list.
  • When you're finished, could you please add a comment to the task about your general impressions? Do you notice any patterns that should be fixed in the algorithm? Do you think this algorithm would do a good job suggesting links to newcomers?

Thanks @MMiller_WMF @DED ! Overall the tool did a good job for the link suggestions. i have two comments:

  • In many cases, the location of the link is not suitable. For example the word ابن خلدون in the article exists twice. But the suggestion is made for the second word.
  • The tool has not made a good recommendation for certain proper names. Especially when names are too long (over three words).

For newcomers the algorithm is a good start for understanding how links work and for exploring articles.

Trizek-WMF triaged this task as Medium priority.Feb 18 2020, 2:37 PM

Okay here are my comments about the algorithm:

  • As @Dyolf77_WMF has mentioned, the suggestion is made not for the first word to appear. But I still marked it done with acceptable word.
  • Many important and specific words are not linked, but the algorithm instead chooses to link very general words that have no reason to be linked.
  • Break words when linking, sometimes they leaves out the words that exist in the article name too ("bệnh ung thư vú", but link only "bệnh ung thư" while we have the article "bệnh ung thư vú"), sometimes it combines and links word in a wrong way ("thỏa mãn tính tò mò" (satisfy (thỏa mãn) the curiousity (tính tò mò), the algorithm combine "mãn" and "tính" (yeah, "mãn tính" has a meaning too) and link it, even though each of the words belong to other words).
  • Link to the wrong article: the phrase and the article are about two entirely difference matters.

With the quality of the algorithm in my language I'm afraid if we introduce it to newcomers they would be more confused and end up getting the wrong idea about linking and might make many articles worse. The algorithm now links a lot of general words, what if newcomers think that it's necessary to link every words in the article too? I think my language is a little too tricky for the algorithm to pick the right phrase to link for now.

Examples for the 2nd and 4th points: in this article, the phrase that should be linked is: gen (gene), chromosome, base (the thing in chemistry), Màng nhân (nuclear envelope), tế bào ung thư (cancer cell)... all these phrases are important in a biology-related article, but the algorithm instead links: "limited" (general) with "limits (in maths)", "position" (general) with "position (in geometry)", "life" (general) with "life (in biology)", "determine" with "determinism". These are all wrong...

Two problems:

  • It should not be linking to articles about the dates (like [[ko:1919년]] / [[ko:1월 1일]]) as the Korean Wikipedia's Manual of Style demands that "if the event cannot be linked from the dates page, it should not be linked to", and since algorithm cannot guess whether it is link-able or not, it should not link.
  • Disambiguation - logic skipped https://ko.wikipedia.org/wiki/%EB%9D%BC%EC%9D%B4%EB%B8%8C%EB%9F%AC%EB%A6%AC and went to [[ko:라이브러리 (컴퓨팅)]].

Are we supposed to find links that should be linked, but weren't marked by the algorithm?

Urbanecm updated the task description. (Show Details)

@Urbanecm -- you do not need to identify links that should be linked. You only should evaluate that ones that were linked by the algorithm. But when you are finished, please post a comment about your general impressions of the algorithm's performance, the patterns you see when it makes mistakes, and some examples of words that should have been linked but were not.

I just went through the Vietnamese, Korean, and Arabic lists to count up how many of the suggestions were good ones. I added those to the task description under each language's header. I also added the count for each article next to its checkbox.

Arabic: 85%
Vietnamese: 52%
Korean: 78%

Done. My thoughts:

  • Algorithm is better than before, many recommended links are good, but it still leaves some important links (add links to one of three names, and leaves other two names without any link).
  • A little amount of links is totally irrelevant, which is easily noticeable by a newbie, but other links are incorrect because of the policies. Our product needs to guide newbies, so experienced users don't complain "the Foundation product causes incorrect edits" :-).

Thank you, ambassadors. This task is complete and helped us decide that we want to continue working with the algorithm.

@DED -- I've now added an evaluation of the English articles to the task description, along with notes for potential improvements.

MMiller_WMF renamed this task from Newcomer tasks: evaluate link recommendation (Feb 14 2020) to Add a link: evaluate link recommendation (Feb 14 2020).May 14 2020, 8:52 PM

@DED Please take time to respect copyright when you import content on testwiki - you forgot to mention that you've copy/pasted it from corresponding Wikipedia pages :)

Thank you for checking out the task, @Framawiki. This is a good reminder about copyright. We do want to keep these pages around -- but I think we can probably add something to the top of each page to explain where the content came from. Do you think that's the right way to handle it?