Bit of a first draft, this defines a few new profiles:

• default - The existing prod deployment
• expensive_1 - Existing suggest field, allow more corrections, generate more candidates, min prefix len of 1
• expensive_2 - Existing suggest field, allow more corrections, generate more candidates, min prefix len of 2
• variant - Default configuration against suggest_variant (opening_text) field
• expensive_1_variant - expensive_1 against the suggest_variant field
• expensive_2_variant - expensive_2 against the suggest_variant field

The queries used here are a completely biased sample I came up with looking at the fulltext head queries dashboard, along with reviewing samples of logged queries that we provided suggestions to. They will be biased towards things i found "interesting", such as bad rewrites by the old engine.

Trying to put some sort of judgement on the list i came up with the following categories:

• IR: Incorrect rewrite of bad query
• FR: Failed to rewrite bad query
• RG: Rewriting a query that was already good
• NG: Not rewriting a good query
• GR: Plausibly good rewrite of bad query

An improved ranker should move queries from IR or FR to GR, and from RG to NG.
If we pivot on that with the above set of queries we get:

Some thoughts:

• The variant field seems to always outperform the default field when using the same configuration. But the response time is also >= 3x.
• NG being 1 (or 0) for the default profile is by construction. Those queries were chosen from query logs as examples where the production suggester gives bad suggestions. It's interesting that the more expensive configurations against the same field don't seem to improve.
• expensive_1 did a decent job of moving some queries from IR to GR while not overly increasing runtime costs. Arguably expensive_1 did a better job on this test set than expensive_2_variant, despite half the mean runtime and not having access to the expanded statistics of the variant field.
• expensive_1_variant shows significant improvements from the baseline default profile, suggesting that if this test set is representative then we have the opportunity to help some number of queries.
• This dataset may not be particularly representative though, it was constructed by hand by selecting "interesting" queries from query logs. The bias in dataset collection limits generalizability, but I think still provides a reasonable place to decide on tests from.
• it looks like our largest wins, and our largest costs, come from reducing prefix length from 2 to 1.

For this set of profiles, I'm thinking:

Profiles to pass on:

• variant - skip in favor of expensive_2_variant
• expensive_2 - skip, no real benefit over default

AB test to run:

• default - existing prod
• expensive_1 - interesting result, good performance without the latency cost of the variant field
• expensive_2_variant - Could go with this or variant. If we pushed variant as the default prod, it would have to be one of these two. Expect to compare against expensive_1.
• expensive_1_variant - Expecting latency cost to be too much to ship, but this was the strongest performer. Seems worth AB testing to see the numbers.

Nice write up! Your conclusions make sense to me. Do you want to do an A/B/C/D test, or a series of (three?) A/B tests?

An improved ranker should move queries from IR or FR to GR, and from RG to NG.

For my own sanity, I reformatted your table. I couldn't keep the abbreviations straight at first so I grouped things that go together together and bolded the "good" columns that we want to see increases in. Makes more sense now, and I can more easily see that *R == IR+FR+GR == 35, and *G == RG+NG == 16.

Note: After talking about this more, we determined that default is equivalent to default_2 and variant is default_2_variant, so I've renamed them to make the relations clearer.

As I mentioned on IRC I'm less concerned about FR so I think moves from IR to FR are good, too—doing nothing is better than being really wrong. RG going down and net moves from IR to FR probably won't change clickthroughs, but they will both help with the reputation and confidence in the suggestions as they increase precision (by decreasing the denominator).

expensive_2 - skip, no real benefit over default

It's actually the same of worse in every column!

The variant option really is doing the heavy lifting.

The variant option really is doing the heavy lifting.

The variant decreases IR by 7 and RG by 8. prefix_1 seems to decrease IR by another 8 (expensive_2 vs expensive_1). So I'd like to see default_1, and default_1_variant. Maybe expensive isn't the most important part (welding defects be damned).

Per discussion at wed meeting i added a couple more profiles and renamed the existing profiles to be more consistent. The names should now consistently be of the format: {profile_name}_{prefix_len}(_variant)?. The numbers are mostly but not directly comparable to above, all the queries were re-run which gave new latency numbers and different results for some queries. All of the results were re-graded into one of the 5 buckets.

The summary might be that the expensive versions cost more, but don't improve metrics much. Sometimes the metrics even go backwards on the more expensive query. At least on this test set. As Trey stated above, the variant field was doing most of the heavy lifting. The default_1_variant, which is the existing production configuration (mostly opensearch defaults) with prefix_len=1 and the opening_text field for token statistics, performs the best on this set of queries.

Curiously the mean latency also went down compared to the previous run, but I'm not sure how meaningful that is. We will likely need to look at latency on the full AB test to have enough data to derive good stats around how much the latency changes.

Based on this, it almost seems the expensive variants are a wash and not worth moving forward to the AB test.

Current proposed AB test:

• default_2 - existing prod configuration
• default_1 - promsing configuration wrt latency, changes only prefix_length from current prod config
• default_1_variant - best performing configuration. If it's truly deployable will depend on latency results. Will also be interesting to know if these improvements lead to measurable changes in behaviour. Even if we cant ship it, maybe we can find a middle ground.

default_1 looks like a winner if default_1_variant turns out to be too expensive. The A/B test results on latency and clicks will be interesting.

Seems like we have a decision. T404647 created to run the test.