@Dbrant How did you evaluate your morelike A/B testing? ( https://phabricator.wikimedia.org/T125393 ) Is it possible to re-use your system for Citolytics?

@mschwarzer The question that I'm hearing is whether we could add this new A/B test to the Android app to evaluate Citolytics suggestions against our current "morelike" suggestions (is that correct?). This can certainly be done, but we'll need a couple additional bits of information:

• What's the time frame within which you'll need these results? (so that our team can prioritize this work accordingly)
• What exact changes need to be made to the API query that gives us the suggestions? Our current query is of the form generator=search&gsrsearch=morelike:foo.

@Dbrant Yes, that's correct!

• The time frame depends on how fast the integration with CirrusSearch can be done. In general, we hope to collect enough data (over 1-2 weeks) before Christmas. But of course, the sooner the better.
• Only the query prefix needs to be changed from morelike: to citolytics:. For all changes please have a look at my GitHub fork: https://github.com/mschwarzer/apps-android-wikipedia/commit/d676eadd330b331065dc27007a8934844a3b97af (only a draft, still WIP)

Cool! In that case, just let us know when the citolytics: query is ready to be used in production. Your change in GitHub looks pretty good! When ready, feel free to submit it as a patch to our Gerrit repo, and we'll be good to go.

If we are hoping to use clickthrough data to measure effects let's make sure we have calculated that metric for a bit before we run the experiment and that is a sound one when it comes to measure the effect we want to measure.

Also, let's think about how are we going to size buckets for A/B testing.

@mschwarzer can you elaborate on the current fuctionality of citolytics and perform the analysis for the current recommendation mechanism. I hope the data you need is availible to the public.

@Physikerwelt The data is currently not available, but I already requested the release (See https://phabricator.wikimedia.org/T125393 ). As soon as the data gets public I'll do the analysis.

What's the status/fate of this task? Can it be closed?