@ovasileva @RHo please see the test results in the description. Nothing surprising here.

Regarding the breakdown of how people submitted their searches, it was a bit different than the actual data we got in T259766#6400566 which was 51% suggested search results, and 48% search button/enter key. In the case of the test it was 86 of 117 (or 73%) of searches submitted via a suggested search result Of course our study is way too small to draw any conclusions but it made me wonder: do we know anything about the current search experience in terms of do-over searches (made up term)? Like do people type "Prince" and hit enter, land on the page about Princes but actually want the page about Prince the musician, then do a second search to get there? Since the search results are now larger, and have images and descriptions, maybe people will be more likely to click them (vs using search button/enter key). If so, perhaps there will be fewer cases where people do do-over searches, as in doing a second search when their first search didn't take them where they wanted to go. From a data perspective this might actually look like people are searching less(?), so I wonder if we want to:

1. make sure that we're not counting do-over searches in the total # of searches conducted that we're using in our A/B tests
2. somehow try to track if the new search experience results in people getting to the page they were looking for more often (i.e. less do-over searches)

In T261515#6420028, @alexhollender wrote:

@ovasileva @RHo please see the test results in the description. Nothing surprising here.

Regarding the breakdown of how people submitted their searches, it was a bit different than the actual data we got in T259766#6400566 which was 51% suggested search results, and 48% search button/enter key. In the case of the test it was 86 of 117 (or 73%) of searches submitted via a suggested search result Of course our study is way too small to draw any conclusions but it made me wonder: do we know anything about the current search experience in terms of do-over searches (made up term)? Like do people type "Prince" and hit enter, land on the page about Princes but actually want the page about Prince the musician, then do a second search to get there? Since the search results are now larger, and have images and descriptions, maybe people will be more likely to click them (vs using search button/enter key). If so, perhaps there will be fewer cases where people do do-over searches, as in doing a second search when their first search didn't take them where they wanted to go. From a data perspective this might actually look like people are searching less(?), so I wonder if we want to:

1. make sure that we're not counting do-over searches in the total # of searches conducted that we're using in our A/B tests
2. somehow try to track if the new search experience results in people getting to the page they were looking for more often (i.e. less do-over searches)

@MNeisler - thoughts on this? I think it would be tricky to define what a do-over search is - I guess we can set some sort of time limit for doing multiple searches with the same search term and look at that? Does the current instrumentation allow for that?

In T261515#6422305, @ovasileva wrote:

@MNeisler - thoughts on this? I think it would be tricky to define what a do-over search is - I guess we can set some sort of time limit for doing multiple searches with the same search term and look at that? Does the current instrumentation allow for that?

@ovasileva @alexhollender
It's possible to track do-over searches as described above but it would be complex and time-intensive to calculate using current instrumentation. We could review search sessions that follow a specific sequence of search actions and have partial search term matches. I'm not sure reviewing just full search term matches in a single search session would provide an accurate estimate - For example, if the person types "Prince" and hits enter, the search query would be recorded as "Prince" but when the person does a second search, they might just start typing "Pri" and then select the desired autocomplete result as soon as it appears. The search query this time would be recorded just as "Pri".

Let me know if you think is important to know for the search experience design and I can create a ticket to investigate some options further.

@MNeisler @ovasileva
Ok, well currently this is totally speculative on my part and I think would only be worth investigating if we think do-over searches are somewhat common. @TJones any chance y'all have thought about, or know about, do-over searches (see above comments for context)?

In T261515#6430489, @alexhollender wrote:

@TJones any chance y'all have thought about, or know about, do-over searches (see above comments for context)?

We have looked at query reformulation within the same session, but only for full text search, not search from the "go" feature.

• The "go" feature is the query that is currently in the upper corner, and which, in the last version I saw, you had moved to the middle of the top of the screen. We generally think of those as successful searches, on the assumption that if people choose a suggestion they are getting what they want—but your example with typing Prince and hitting enter is a good potential counter example.
• We have defined search sessions as the same IP, user agent, etc. with a gap between queries of less than 10 or 20 minutes (I think we've used both). So, if someone walks away from their computer for an hour and comes back, that's definitely a new session, but searching every 5 minutes for an hour is one session.
• Accurate detection of query reformulation (which sounds like your "do-overs") is hard. In the "go" box it's even harder. If you type slowly (and poorly) but have a fast internet connection you could log all of the following queries to the completion suggester: p, pr, pri, pric, pricn, pricne, pricn, pric, princ, prince. If you only log what's in the text box when they hit enter or select a suggestion, it's still hard, especially because of the pri → Prince (musician) example above.
  • "Real" reformulation happens when there's some textual similarity between queries—pricne and prince or dogsled and dog sled or soccer finals and football finals. Note that small edits like pricne and prince don't make as much sense in Chinese, though, where changing one or two characters can completely change the meaning. So, for machine learning purposes, you can just count all queries in one session as "reformulations" even when they aren't, and the ones that recurr often are probably the "real" ones and you can learn from those. For full-text A/B tests, fewer reformulations probably means that searches are more successful, because there are fewer "real" reformulations... but you never get rid of people searching for totally different things in the same session. I think you could similarly interpret the final text in the "go" box. (Though A/B tests in search are always noisy.)

@EBernhardson has more experience with all of this, but that should cover most of the basic stuff we've thought about.