Page MenuHomePhabricator
Create Task
Maniphest T267971

Analyze Speaker-Reviewed M2 Data for Chinese
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

User Story: As a search developer, I want to have some idea of how Glent M‌2 is performing on Chinese queries so that I can make informed decisions about next steps (such as making improvements to Glent or running an A/B test).

Notes: The data has been reviewed by the speaker but the final analysis hasn't been done yet. The goal of the parent task was to do one or more of the CJK languages, and Japanese and Korean were done and the analysis reviewed. We still need to do Chinese and decide what to do there.

Acceptance Criteria:

  • Write up is completed and reviewed by at least one other search engineer.
  • Next steps are decided (make improvements, do an A/B test, etc.) based on recommendations.

Related Objects
Search...

Event Timeline

TJones created this task.Nov 16 2020, 8:49 PM
Restricted Application added a subscriber: revi. · View Herald TranscriptNov 16 2020, 8:49 PM
TJones renamed this task from Review Chinese M2 Data to Analyze Speaker-Reviewed M2 Data for Chinese.Nov 16 2020, 8:49 PM
TJones edited projects, added Discovery-Search; removed Discovery-Search (Current work).
TJones updated the task description. (Show Details)
CBogen triaged this task as High priority.Nov 23 2020, 6:35 PM
CBogen moved this task from needs triage to Language Stuff on the Discovery-Search board.
Shizhao moved this task from Backlog to Research on the Chinese-Sites board.Nov 24 2020, 4:05 AM
TJones edited projects, added Discovery-Search (Current work); removed Discovery-Search.Feb 8 2021, 4:29 PM
MPhamWMF set the point value for this task to 3.Feb 8 2021, 4:45 PM
MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
TJones claimed this task.Feb 11 2021, 6:14 PM
TJones moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.
TJones added a comment.Feb 24 2021, 10:05 PM

Summary: The stats for Glent M2 suggestions for Chinese are roughly similar to Korean and Japanese. A big difference is that a fair number of suggestions are traditional-to-simplified conversions, which were rated as good suggestions, but which probably don't make much difference in search results (we do traditional-to-simplified conversion for indexing and searching behind the scenes)—though it is possible that Glent's traditional-to-simplified conversion is better for these queries than our rule-based one for searching and indexing.

Chinese production DYM also has very few suggestions for actual Chinese text (as opposed to suggestions for Latin/English text)—only ~5% of suggestions are Chinese.

As before, the new Glent M2 suggestions are largely orthogonal to the production DYM suggestions, so an A/B test seems like the right next step—though not until T265081 is done.

Full write up is on Mediawiki.

TJones moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Feb 24 2021, 10:06 PM
dcausse added a subscriber: dcausse.Mar 4 2021, 5:18 PM

Great write-up thanks!

TJones moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Mar 4 2021, 6:23 PM

Thanks for taking a look!

Gehel closed this task as Resolved.Mar 8 2021, 1:06 PM
Shizhao moved this task from Research to Closed on the Chinese-Sites board.Mar 10 2021, 8:46 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL