Page MenuHomePhabricator

Write Glent M0 A/B test report
Open, Needs TriagePublic

Event Timeline

dcausse created this task.Nov 5 2019, 10:02 AM
Restricted Application edited projects, added Discovery-Search; removed Discovery-Search (Current work). · View Herald TranscriptNov 5 2019, 10:02 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

While reviewing the related code we found a bug in the UI of the provided suggestions that essentially invalidates the AB testing done. Essentially the UI was misrepresenting which query was run, discouraging users from clicking the suggested query (and generally being hard to interperet). The fix[1] has been merged and will deploy with the next train. The test should be re-run once fixed and the suggestions pipeline has been verified.

[1] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CirrusSearch/+/554603/

This ended up further delayed due to not being able to differentiate queries suggested by glent from queries suggested in general. Essentially glent does not suggest queries often enough to make a measurable impact when mixed in with the other suggestions. We expect that by logging which queries were suggested by glent we can compare the metrics on those queries vs the metrics on our normal suggester, and have better data. The data collection update has shipped and we should be turning the test back on soon.