Page MenuHomePhabricator

Newcomer tasks: ambassadors test topic matching in production
Closed, ResolvedPublic

Description

It is time to test out topic matching in production. It is currently only available to users who run the code below to turn on the hidden preference. We want to have all test results by Monday, Jan 20 so we can plan to expose the feature to newcomers on Tuesday, Jan 21. Below are instructions for how to test.

  • Czech
  • Korean
  • Vietnamese
  • Arabic
  1. Go to the newcomer homepage in your wiki.
  2. Reset the suggested edits workflow for your user. Do this by opening your browser tools (by right-clicking and choosing "Inspect") and running this in the console: new mw.Api().saveOption('growthexperiments-homepage-suggestededits-activated', 0).done( function() { window.location.reload() });
  3. Enable topic matching by running this in the console: new mw.Api().saveOption('growthexperiments-homepage-suggestededits-topics-enabled', 1).done( function() { window.location.reload() });
  4. Go through the workflow of initiating suggested edits, choosing topics, and changing topics. Try everything out! You can always re-run Step 2 to reset the whole workflow and see the initiation pieces again.
  5. Note any issues relating to:
    • Missing or incorrect translations
    • Interface behaving strangely or looking weird
    • When you select a topic, get results, then switch topics and switch back, you are supposed to get different results than you got the first time. We want to make sure that not all users who select "Music" get the same article suggestions.
    • Poor topic results. This is important, especially because @Dyolf77_WMF noted in T242400 that results in the module looked different than the ones coming over the API. Please try out several different topics and think about whether the performance is better or worse than previous tests.
  6. Try all of this on your mobile device. You can reset the workflow on desktop with Step 2 so that you can try it on mobile.

Please post any issues or concerns as comments on this task. If you have to update translations, let us know when you've done that in Translatewiki so that @Catrope can update them manually before the next train runs.

Event Timeline

MMiller_WMF moved this task from Incoming to In Progress on the Growth-Team (Current Sprint) board.
MMiller_WMF updated the task description. (Show Details)Jan 16 2020, 9:52 PM

For completeness: if you want to turn topic matching back off and test the UI that normal users see, you can run step 3 but with the 1 changed to a 0:

new mw.Api().saveOption('growthexperiments-homepage-suggestededits-topics-enabled', 0).done( function() { window.location.reload() });
Dyolf77_WMF moved this task from Backlog to In progress on the User-Dyolf77 board.

Arabic Wikipedia:

  • No missing translations;
  • The interface seems to be correct;
  • Different list after re-selecting topics;
  • Tested the SE module (both: Desktop and Mobile) in ar wiki, same observations as in beta: 1 to 3 less good articles compared to the test made on the API.
PPham added a comment.EditedJan 18 2020, 5:02 PM

Vietnamese Wikipedia:

  • Both desktop and mobile version are still missing some translations. Also I have some translations I want to change, can I?
  • The result articles are off-topic by half. I'm working on my seed article list again, please wait...
  • I have just realized that some times along the way the Vietnamese seed article list have become incorrect: In the Json list, the Science tab has now been labeled "Địa lý" (Geography) with Geography's seed articles. So now (1) there's no science label and science seed articles; (2) there's no geography tab.

I have some thoughts about the topics: they are not "equal". I mean, some are very general (science, art, education...) but some really go into detail. For example, art includes both music and films; Government is just a sub-topic of Politics; The same goes with business and economy... So you understand my point. I know that these topics are only temporary, I just wanna give some serious thoughts about them as they are an important part of our prj.

revi added a comment.Jan 19 2020, 12:34 PM

kowiki;

  • Few hotfixes (including missing translations) deployed (Revision 25588448, 25588451, 25588609, 25588628, 25588634, 25588641, 25588667, 25588680) - I have a reminder to delete them once the train has stopped at Korean Wikipedia station :P
  • I get different results for re-try. Some topic has less article (i.e. biography has 83 to draw from), so it might be more likely that people will notice result reuse if they try getting list quite often, though.
  • Quality seems to have significantly dropped: at least 3 mismatches in all topics. (Geography got 0 correct articles in first 10, history got 4/10, music 4/10, Sports 2/10, literature 2/10, religion 2/10, pop culture 4/10, entertainment 5/10, foods 2/10, biography 4/10, military 7/10, economy 3/10, )
Urbanecm updated the task description. (Show Details)Jan 19 2020, 6:39 PM

I spent few hours playing with the interface, and I'm now satisfied with its average performance. @Tgr explained in ambassadors' chat why the results are worse - the API link we were playing with was giving top ten matches, randomized in order, while homepage has top 250 matches, randomized in order. That's always going to be worse. Checked "Czech", please let me know if I should do anything else.

Thank you, all! Now we have a good understanding of how much effect the randomization has on the strength of the topic matches. We'll talk with the engineers about what, if anything, can be improved.

@PPham -- please make any translations and updates you want in Translatewiki, and let us know when they're there. Then we can move them to your wiki.

Trizek-WMF triaged this task as Medium priority.Jan 20 2020, 6:51 PM
Trizek-WMF updated the task description. (Show Details)Jan 29 2020, 3:35 PM
PPham added a comment.Feb 1 2020, 2:39 PM

@MMiller_WMF It's okay now, please help me move them. Sorry for being this late...

Trizek-WMF closed this task as Resolved.Feb 3 2020, 8:30 AM
Trizek-WMF updated the task description. (Show Details)
Urbanecm edited subscribers, added: Urbanecm_WMF; removed: Urbanecm.Aug 26 2020, 3:24 PM