Page MenuHomePhabricator

Newcomer tasks: evaluate topic matching prototypes (ar)
Closed, ResolvedPublic

Description

This is a task for @Dyolf77_WMF to complete the work described in T234272.

Details

Due Date
Oct 15 2019, 1:00 PM

Event Timeline

@Dyolf77_WMF -- only the first two methods are available for Arabic right now. @kostajh is still finishing up ORES, and will let us know when it is ready.

MMiller_WMF set Due Date to Oct 15 2019, 7:00 AM.Oct 1 2019, 4:34 PM
Dyolf77_WMF triaged this task as High priority.Oct 1 2019, 4:58 PM

Hi @Dyolf77_WMF I am moving this to Ready for Development on the Growth team Board, feel free to move this to In Progress when you start working on it

Trizek-WMF changed Due Date from Oct 15 2019, 7:00 AM to Oct 15 2019, 1:00 PM.Oct 2 2019, 2:48 PM

@Dyolf77_WMF -- I wanted to let you know that the ORES prototype is now ready to evaluate, so the entire set of evaluations is ready for you to work on.

@Dyolf77_WMF -- I wanted to let you know that the ORES prototype is now ready to evaluate, so the entire set of evaluations is ready for you to work on.

Thanks Marshall, the evaluation will be done by Tuesday.

Dyolf77_WMF closed this task as Resolved.Oct 7 2019, 9:30 PM

Thanks, @Dyolf77_WMF. I have two additional requests:

  • Could you please add the English translations of your "free text" items on the "free text" tab? I added a column for it.
  • Could you comment on your general impressions? Which method worked best? What are the pros and cons of the different methods? What do you think will be best for newcomers?
Dyolf77_WMF reopened this task as Open.Oct 8 2019, 8:13 AM
Dyolf77_WMF added a comment.EditedOct 8 2019, 9:17 AM

Hi @MMiller_WMF

  • Could you please add the English translations of your "free text" items on the "free text" tab? I added a column for it.

Done

  • Could you comment on your general impressions? Which method worked best? What are the pros and cons of the different methods? What do you think will be best for newcomers?
  • General impressions: The three methods are doing an interesting work, but we have to consider the target wiki and especially the quality and the amount of articles there; I felt that these methods of topic matching will work more better if there is enough articles to recommend. I had many articles appearing again and again for different topics. I presume that there is not enough ressources these tools are fetching in. About the quality, I have a concern about the reaction of newbie facing a very bad written article with a lot of errors.

You will keep fishing salmons in a river instead of tuna.

  • The score of the ORES method is the best 77.0%, followed by the "more like" 58.8% and in the last rank the "free text" method with 47.3%. Also, I found the ORES method is finding better matching articles.
  • More like: could work better if there is enough articles, seems to repeat the same articles (out of topic).
  • Free text: working well for general topics but not doing well with specific ones. Maybe should I add some other topics and see the results?
  • ORES: why low scores for some topics, that's because as the More like method, it repeats some articles from topic to topic. Otherwise, I am very impressed about the quality of the research with this method.
  • As the statistics are showing and after seeing all the results, the ORES method is the better for newcomers.
Dyolf77_WMF closed this task as Resolved.Oct 8 2019, 9:18 AM
Urbanecm edited subscribers, added: Urbanecm_WMF; removed: Urbanecm.Aug 26 2020, 2:10 PM