Page MenuHomePhabricator

Perform A/B test to determine if using opening_text instead of text as the field to perform more_like_this queries is better or not
Closed, DuplicatePublic

Description

more like this queries are the least performant queries on the cluster, and despite being ~5% of the total query volume take up more than 20% of server resources.

We should be able to measure the impact, in terms of server load, by reusing the load testing that was put together for T117714. gor allows middleware to be written in any language (python is readily available on elastic servers) to pre-process the request before sending it. This middleware can rewrite the more like this queries and we can then measure the difference in server load and any increase in throughput.

We also need to determine if the results are any better/worse like this. We can initially run a number of these through the relevance lab to see that there are any differences. Next we can run an AB test measuring user click through rates to see if users like the new suggestions any better than the old ones.

Currently any request that wants to try the new results here can append the following uri query parameters to get the new functionality:

cirrusMltUseFields=yes&cirrusMltFields=opening_text

This fits nicely into one of our Q3 goals, evaluating current cirrus options and adjusting to what makes the most sense.

Event Timeline

EBernhardson raised the priority of this task from to Needs Triage.
EBernhardson updated the task description. (Show Details)
EBernhardson added a project: CirrusSearch.
EBernhardson added subscribers: EBernhardson, dcausse.
Restricted Application added a project: Discovery. · View Herald TranscriptJan 21 2016, 1:03 AM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

Related task T124216 is about caching these morelike results

Deskana renamed this task from Use opening_text instead of text as the field to perform more_like_this queries against to Perform A/B test to determine if using opening_text instead of text as the field to perform more_like_this queries is better or not.Jan 21 2016, 11:13 PM
Deskana set Security to None.
Deskana added a subscriber: Deskana.

Renamed this task to clarify that the intention here is to perform an A/B test to determine whether or not this is really better. Our quick, gut-check analysis indicates that it is simply better (irrespective of performance improvements), but we should really check that properly, shouldn't we? :-)

I'm going assume yes, and merge the tasks.

debt moved this task from Needs triage to UX on the Discovery board.Feb 4 2016, 8:56 PM