Samples are available here: notebook1004:/home/dcausse/phrase_suggester_vs_glent_m1.csv

I completed my analysis of Method 1, and it performs significantly worse than the current production DYM. I think we should improve Method 1 before considering an A/B test. Full details on MediaWiki.

Summary:

• When Method 1 and production DYM disagree, Method 1 was better than prod 7% of the time, both were good about 26% of the time, and prod was better 40% of the time.
• Analyzed alone, prod suggestions were poor 45% of the time (which is why we are here!), but Method 1 suggestions were poor 72% of the time; when they agree, only 23% of suggestions were poor.

Method 1 Anti-Patterns:

• over-emphasis of result counts—
  • creating negated queries, like fogus to -ous which gets 5.9M results.
  • changing letters or adding spaces to create a very common word (cf gene to a gene) or duplicated word (rattle battle to battle battle).
• overly drastic changes—
  • edit distance limits should be per-token, not per string (cf gene to a gene again)
  • changing a letter to space should have a higher cost (abbys to a b s)
  • changing the first letter of a word/token should have a higher cost (cia assassinations to mi6 assassinations)
• using weird stemming edge cases to increase result counts—
  • e.g., godness stems to god so it beats goddess; hering stems to here so it replaced herring in red herring

Reinforcing Positive Method 1 Patterns:

• Edit distance cost should be decreased for double-letter to single-letter change (or vice versa)
• Edit distance cost should be decreased for swapped letters, possibly including swapped with a letter in between (levasimole vs levamisole)

I realize that I've assumed that edit distance plays a role in the weighting of suggestions, but I'm not sure that's the case. If not, it probably should be, rather than letting result count reign supreme.