In T231506, we explored several methods with which to surface articles to newcomers based on the topical interests of those newcomers. This is difficult because newcomers have no editing history with which to make recommendations.
This task is about evaluating three methods that we've put into interactive prototypes. Below, we describe each of the prototypes and how ambassadors can evaluate them.
1. Morelike
- Prototype
- How it works: Newcomer selects from a list of 27 broad topics. Each of those 27 topics has a corresponding list of articles that are pre-set by ambassadors in T233465 (the "seed" articles). For each of the topics that the newcomer selects, the prototype takes the seed articles and does a search for more articles that have a lot of the same words in common with the seed articles. It narrows the results to those that have a maintenance template and displays the results.
- How to evaluate:
- Select your language.
- For each of the 27 topics...
- Select the topic.
- Select all the task type checkboxes.
- Leave all other settings alone.
- Look at the first ten articles that get returned and count how many of the ten are good results for that topic. For instance, the article on "Elevator" would be a good result for the topic "Engineering". But the article on "Shoes" would not.
- Write down that score in this sheet.
- If any topic has fewer than 10 results, indicate that by making a note in the cell.
- Notes:
- You can click each result to see details about its templates, categories, and the search that was run.
- The prototype contains some additional algorithm settings. You are welcome to play with those and record some of your notes about what you notice, but we're evaluating them based on the default settings.
- Although the prototype allows you to select only certain maintenance templates, we think you should select all of them for this exercise, because we're really only evaluating the topic matching abilities here. We can separately count how many results show up for each maintenance template.
2. Free text
- Prototype (same as morelike)
- How it works: Newcomer types in some text in the search field, and it runs a normal search, just like the search bar in Wikipedia, but narrowed to articles with maintenance templates. This allows the user to search for more specific topics.
- How to evaluate:
- Select your language.
- Select all the task type checkboxes.
- Leave all the other settings alone.
- Type one topic at a time into the free text field. Please try 15 different topics of your choice, that can more or less specific. A more general one might be "Swimming" and a more specific one might be "Pokemon".
- Look at the first ten results for that search term, and count how many look like good results.
- Put the search term and the score in this sheet.
3. ORES
- Prototype
- How it works: There is a machine learning model in English Wikipedia that classifies any English article into a topic. The topics are made through the English WikiProject hierarchy and are not the same as the ones from the "morelike" list (but we could align them later if we like this approach). The method takes all the articles with maintenance templates in the target wiki, then finds the ones that also exist in English Wikipedia, then gets their ORES topic score from English and applies it to the target language's version. That means that the only articles that come up are the ones that exist in English, too. That's not optimal because it would mean that we don't recommend any local-language articles for editing, but we still want to try this method out to see how good it is.
- How to evaluate:
- Select your language.
- For each of the 42 topics...
- Select the topic.
- Select all the task type checkboxes.
- Look at the first ten articles that get returned and count how many of the ten are good results for that topic.
- Write down that score in this sheet.
- If any topic has fewer than 10 results, indicate that by making a note in the cell.
- Notes:
- We only have the English names of the topics, but if we like this method, we would figure out how to translate them to local languages.