Page MenuHomePhabricator

Beta A/B test of new search auto-completion generator.
Closed, ResolvedPublic2 Estimated Story Points

Description

The new auto-completion functionality from the Discovery-ARCHIVED team has been integrated as a generator in the API (T121430), which means that we can plug it in when performing searches from our Search bar. We can now do an A/B test of the old prefixsearch vs. the new autocompletion, and measure the resulting clickthrough rate.

  • This should be feature-flagged for Beta only, since the generator itself is still experimental.
  • No changes needed in the EL funnel, AFAICT. Just switch out the generator!

Event Timeline

Dbrant raised the priority of this task from to Needs Triage.
Dbrant updated the task description. (Show Details)
Dbrant moved this task to Next Sprint on the Wikipedia-Android-App-Backlog board.
Dbrant subscribed.
Dbrant set Security to None.

Change 271304 had a related patch set uploaded (by Mholloway):
Implement A/B test for beta search autocomplete feature

https://gerrit.wikimedia.org/r/271304

Change 271304 merged by jenkins-bot:
Implement A/B test for beta search autocomplete feature

https://gerrit.wikimedia.org/r/271304

Checked the all search test cases - Boolean, exclusion, wildcards, prefixes(special prefixes) - nothing jumped out as incorrect.
Looked at T121616: EPIC: Resolve technical or product blockers to enable us to do a more full rollout of the completion suggester - to check if there are relevant un-fixed cases.
Looked at T122932: CompletionSuggester prefers perfect redirect match over near-perfect target match - it's fixed in my opinion.

Only two things - quite minor in my opinion.:

  • with Beta feature enabled - the search results for search test cases as *end and *begin differs from when the same search is made with the disabled feature. But it's difficult to evaluate which search is more relevant in this case.

Screenshot_2016-02-19-13-46-38.png (1×1 px, 289 KB)

Screenshot_2016-02-19-13-46-52.png (1×1 px, 227 KB)

Clear the search results, if any. Disable the beta feature and force the app to stop. Re-launch and re-enter *end and *begin as search criteria.

Screenshot_2016-02-19-13-13-27.png (1×1 px, 258 KB)

Screenshot_2016-02-19-13-13-47.png (1×1 px, 292 KB)

  • Searching for 'mexico city public transportation' resulted with search results in slightly different positions on the Search list depending whether the beta feature was enabled or not.