Page MenuHomePhabricator

[EPIC-ish][Milestone 0] Implement NLP Search Suggestion Method 0 for English
Open, NormalPublic

Description

This ticket is to track work by @Julia.glen and others on implementing NLP Search Suggestion “Method 0” for English.

From the parent task:

  • Method 0: Mine search logs for query + correction pairs and create an efficient method for choosing candidates to make suggestions for incoming queries based on similarity to the original query, number of results, and query frequency. Only applicable for languages/projects with sufficient search traffic.

Please create sub-tasks or add details here as necessary.

Event Timeline

TJones triaged this task as Normal priority.Jan 3 2019, 8:01 PM
TJones created this task.
debt moved this task from This Quarter to [epic] on the Discovery-Search board.Jan 29 2019, 6:50 PM

Change 489480 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[search/glent@master] Wrap Method Zero into spark

https://gerrit.wikimedia.org/r/489480

Change 489480 merged by DCausse:
[search/glent@master] Wrap Method Zero into spark

https://gerrit.wikimedia.org/r/489480

Change 491657 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] Bug: T212888

https://gerrit.wikimedia.org/r/491657

Change 491658 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] Bug: T212888

https://gerrit.wikimedia.org/r/491658

Change 491657 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] merge of julia.glen and erik.b' code

https://gerrit.wikimedia.org/r/491657

Change 491658 abandoned by EBernhardson:
Bug: T212888

Reason:
squashed into previous commit

https://gerrit.wikimedia.org/r/491658

Change 491657 merged by jenkins-bot:
[search/glent@master] merge of julia.glen and erik.b' code

https://gerrit.wikimedia.org/r/491657

Change 495934 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] Method0 WIP

https://gerrit.wikimedia.org/r/495934

Change 495984 had a related patch set uploaded (by Julia.glen; owner: Julia.glen):
[search/glent@master] Method0 WIP P2

https://gerrit.wikimedia.org/r/495984

Change 495984 abandoned by EBernhardson:
Method0 WIP P2

Reason:
squashed into I2bae8ca9c

https://gerrit.wikimedia.org/r/495984

Change 495934 had a related patch set uploaded (by EBernhardson; owner: Julia.glen):
[search/glent@master] Method0 WIP

https://gerrit.wikimedia.org/r/495934

TJones renamed this task from [EPIC-ish][Milestone 1] Implement NLP Search Suggestion Method 0 for English to [EPIC-ish][Milestone 0] Implement NLP Search Suggestion Method 0 for English.Mar 20 2019, 3:44 PM

Change 495934 merged by jenkins-bot:
[search/glent@master] Method0 WIP

https://gerrit.wikimedia.org/r/495934

An analysis and comparison of M0 and the current DYM suggester is on Mediawiki.

Summary and Suggestions

M0 is clearly much higher precision than the current DYM suggester, though it with limited input data, it doesn’t have as much coverage as one would like.

Clearly it’s ready for an A/B test of some sort (if we feel that is still necessary—I’m happy with the overwhelming improvement it gives—68% better, 5.5% worse, 14% equivalent, 12% unclear).

However, we could still improve its performance in a few easy and not-so-easy ways:

  • We should filter more aggressively on queries that are all numbers, spaces, and punctuation.
  • We should remove the most common invisible characters (except for non-breaking spaces, which should just be regular spaces).
  • We might want to modify M0’s edit distance algorithm to assign a lower cost to swaps. Adjacent swaps are easy (“be” for “eb” in “ebatles”), while longer-distance swaps (“shamaz” vs “shazam”) are harder to detect, but would give a higher weight to such potential self-corrections. These kinds of errors make sense linguistically, too; when people do it in speech it’s called metathesis.
  • Long term, it might be helpful to track self-corrections that come from people clicking on the current DYM suggestions (or, when M0 or others are deployed, any suggestions). At least for current DYM suggestions, these are often lower quality, and I think many people are clicking on them without carefully assessing them.
  • We should work on figuring out how/whether we can increase the time-depth of our training data. Two thoughts come to mind: holding on to the self-correction data longer than 3 months, and holding on to particularly useful self-correction data (i.e., that which was used to make a suggestion that got clicked on one or more times) for longer than 3 months. Though the concerns in both cases may be the same, so the distinction may not matter.

Other thoughts:

  • There may be an error in our logging or in the extraction script, but the query template:discpute gets a current DYM suggestion of template:dispute, but only dispute is recorded in this corpus. I don’t think this is very important for this analysis because M0 doesn’t make any suggestions for queries with colons in them, but something weird is happening.