Page MenuHomePhabricator
Create Task
Maniphest T212888

Implement NLP Search Suggestion Method 0 for English
Closed, ResolvedPublic
Actions

Description

This ticket is to track work by @Julia.glen and others on implementing NLP Search Suggestion “Method 0” for English.

From the parent task:

  • Method 0: Mine search logs for query + correction pairs and create an efficient method for choosing candidates to make suggestions for incoming queries based on similarity to the original query, number of results, and query frequency. Only applicable for languages/projects with sufficient search traffic.

Please create sub-tasks or add details here as necessary.

Details

ProjectBranchLines +/-Subject
search/glentmaster+1 K -321Method0 WIP
search/glentmaster+60 -20Method0 WIP P2
search/glentmaster+684 -512merge of julia.glen and erik.b' code
search/glentmaster+1 K -622Wrap Method Zero into spark
Customize query in gerrit

Related Objects
Search...

Event Timeline

TJones triaged this task as Medium priority.Jan 3 2019, 8:01 PM
TJones created this task.
TJones mentioned this in T212884: [EPIC] Improve Search Suggestions with NLP (Did You Mean / Glent).Jan 3 2019, 8:07 PM
TJones edited projects, added Discovery-Search; removed Discovery-Search (Current work).Jan 3 2019, 8:09 PM
• EBjune moved this task from needs triage to This Quarter on the Discovery-Search board.Jan 17 2019, 6:22 PM
debt moved this task from This Quarter to [epic] on the Discovery-Search board.Jan 29 2019, 6:50 PM
TJones mentioned this in T212889: [EPIC-ish][Milestone 1] Implement NLP Search Suggestion Method 1 for 10 languages.Feb 11 2019, 2:47 PM
gerritbot added a subscriber: gerritbot.Feb 11 2019, 11:05 PM

Change 489480 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[search/glent@master] Wrap Method Zero into spark

https://gerrit.wikimedia.org/r/489480

gerritbot added a project: Patch-For-Review.Feb 11 2019, 11:05 PM
gerritbot added a comment.Feb 12 2019, 9:05 AM

Change 489480 merged by DCausse:
[search/glent@master] Wrap Method Zero into spark

https://gerrit.wikimedia.org/r/489480

gerritbot added a comment.Feb 20 2019, 12:10 AM

Change 491657 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] Bug: T212888

https://gerrit.wikimedia.org/r/491657

gerritbot added a comment.Feb 20 2019, 12:11 AM

Change 491658 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] Bug: T212888

https://gerrit.wikimedia.org/r/491658

gerritbot added a comment.Feb 25 2019, 6:15 PM

Change 491657 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] merge of julia.glen and erik.b' code

https://gerrit.wikimedia.org/r/491657

gerritbot added a comment.Feb 25 2019, 6:16 PM

Change 491658 abandoned by EBernhardson:
Bug: T212888

Reason:
squashed into previous commit

https://gerrit.wikimedia.org/r/491658

gerritbot added a comment.Mar 1 2019, 9:31 PM

Change 491657 merged by jenkins-bot:
[search/glent@master] merge of julia.glen and erik.b' code

https://gerrit.wikimedia.org/r/491657

gerritbot added a comment.Mar 12 2019, 4:47 PM

Change 495934 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] Method0 WIP

https://gerrit.wikimedia.org/r/495934

gerritbot added a comment.Mar 12 2019, 7:50 PM

Change 495984 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] Method0 WIP P2

https://gerrit.wikimedia.org/r/495984

gerritbot added a comment.Mar 13 2019, 8:16 PM

Change 495984 abandoned by EBernhardson:
Method0 WIP P2

Reason:
squashed into I2bae8ca9c

https://gerrit.wikimedia.org/r/495984

gerritbot added a comment.Mar 19 2019, 7:19 AM

Change 495934 had a related patch set uploaded (by EBernhardson; owner: Julia.glen):
[search/glent@master] Method0 WIP

https://gerrit.wikimedia.org/r/495934

TJones renamed this task from [EPIC-ish][Milestone 1] Implement NLP Search Suggestion Method 0 for English to [EPIC-ish][Milestone 0] Implement NLP Search Suggestion Method 0 for English.Mar 20 2019, 3:44 PM
gerritbot added a comment.Mar 20 2019, 8:07 PM

Change 495934 merged by jenkins-bot:
[search/glent@master] Method0 WIP

https://gerrit.wikimedia.org/r/495934

dcausse mentioned this in T221621: Allow FallbackMethod to create their own SearchQuery.Apr 23 2019, 12:34 PM
dcausse mentioned this in T222152: Make fallback methods configurable.Apr 30 2019, 9:55 AM
dcausse mentioned this in T222652: Create a fallback method based on a generic index.May 6 2019, 6:51 PM
dcausse mentioned this in T224494: Add maint script to import data from swift to elasticsearch.May 28 2019, 4:14 PM
TJones added a comment.May 31 2019, 8:42 PM

An analysis and comparison of M0 and the current DYM suggester is on Mediawiki.

Summary and Suggestions

M0 is clearly much higher precision than the current DYM suggester, though it with limited input data, it doesn’t have as much coverage as one would like.

Clearly it’s ready for an A/B test of some sort (if we feel that is still necessary—I’m happy with the overwhelming improvement it gives—68% better, 5.5% worse, 14% equivalent, 12% unclear).

However, we could still improve its performance in a few easy and not-so-easy ways:

  • We should filter more aggressively on queries that are all numbers, spaces, and punctuation.
  • We should remove the most common invisible characters (except for non-breaking spaces, which should just be regular spaces).
  • We might want to modify M0’s edit distance algorithm to assign a lower cost to swaps. Adjacent swaps are easy (“be” for “eb” in “ebatles”), while longer-distance swaps (“shamaz” vs “shazam”) are harder to detect, but would give a higher weight to such potential self-corrections. These kinds of errors make sense linguistically, too; when people do it in speech it’s called metathesis.
  • Long term, it might be helpful to track self-corrections that come from people clicking on the current DYM suggestions (or, when M0 or others are deployed, any suggestions). At least for current DYM suggestions, these are often lower quality, and I think many people are clicking on them without carefully assessing them.
  • We should work on figuring out how/whether we can increase the time-depth of our training data. Two thoughts come to mind: holding on to the self-correction data longer than 3 months, and holding on to particularly useful self-correction data (i.e., that which was used to make a suggestion that got clicked on one or more times) for longer than 3 months. Though the concerns in both cases may be the same, so the distinction may not matter.

Other thoughts:

  • There may be an error in our logging or in the extraction script, but the query template:discpute gets a current DYM suggestion of template:dispute, but only dispute is recorded in this corpus. I don’t think this is very important for this analysis because M0 doesn’t make any suggestions for queries with colons in them, but something weird is happening.
dcausse added a subtask: T233211: Deploy A/B Test for Glent Method 0.Sep 18 2019, 12:58 PM
dcausse closed subtask T233211: Deploy A/B Test for Glent Method 0 as Resolved.Oct 7 2019, 10:04 AM
Gehel added a parent task: T235829: Glent method 0 (session reformulation) A/B tested and deployed by end of Q3.Oct 18 2019, 9:22 AM
dcausse added subtasks: T237363: Undeploy Glent M0 A/B test, T237364: Write Glent M0 A/B test report, T237365: Enable Glent M0 on de, en and fr wikipedias.Nov 5 2019, 10:04 AM
TJones added a subtask: T237550: Increase Glent M0 retention.Nov 12 2019, 8:48 PM
TJones closed subtask T237550: Increase Glent M0 retention as Resolved.Nov 20 2019, 4:56 PM
TJones closed subtask T237363: Undeploy Glent M0 A/B test as Resolved.Feb 12 2020, 4:35 PM
TJones closed subtask T237365: Enable Glent M0 on de, en and fr wikipedias as Resolved.
Gehel closed subtask T246947: Re-run glent m0 AB test as Resolved.Jul 13 2020, 12:51 PM
Gehel moved this task from [epic] to Current work on the Discovery-Search board.Sep 9 2020, 2:46 PM
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
Gehel moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.Sep 14 2020, 5:24 PM
Gehel moved this task from In Progress to Epics on the Discovery-Search (Current work) board.
CBogen added a subscriber: CBogen.Oct 6 2020, 2:21 PM
Gehel renamed this task from [EPIC-ish][Milestone 0] Implement NLP Search Suggestion Method 0 for English to Implement NLP Search Suggestion Method 0 for English.Oct 6 2020, 2:22 PM
Gehel removed a project: Epic.
Gehel moved this task from Epics to In Progress on the Discovery-Search (Current work) board.
TJones moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Oct 26 2020, 6:14 PM

All child tasks are either complete or in "Needs Reporting", so we're mostly done except for the talking about it!

Gehel closed this task as Resolved.Nov 9 2020, 1:05 PM
Gehel claimed this task.
Gehel closed subtask T237364: Write Glent M0 A/B test report as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL