This A/B test will modify our current "read more" request to use these additional parameters:
cirrusMltUseFields=yes&cirrusMltFields=opening_text
We'll need to measure:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Implement A/B test to measure CirrusSearch "opening_text" performance. | apps/android/wikipedia | master | +49 -10 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Deskana | T125603 EPIC: Review current ElasticSearch configuration, and use relevance lab to run tests to optimise the configuration to improve search result relevance | |||
| Resolved | Dbrant | T125393 Implement A/B test to measure CirrusSearch "opening_text" performance. |
I strongly suggest to add cirrusBoostLinks=no to the list of params. The full param list should be:
cirrusMltUseFields=yes&cirrusMltFields=opening_text&cirrusBoostLinks=no
Change 267695 had a related patch set uploaded (by Dbrant):
Implement A/B test to measure CirrusSearch "opening_text" performance.
Change 267695 merged by jenkins-bot:
Implement A/B test to measure CirrusSearch "opening_text" performance.
Did general search regression/comparison testing with 2.1.141-alpha-2016-02-12 build.
Is the outcome (raw data/evaluation) of the A/B test still available? We would like to use it as reference for our Citolytics A/B test.
It looks like this wasn't a super rigorous study, but the rolled up data is at https://docs.google.com/spreadsheets/d/1BFsrAcPgexQyNVemmJ3k3IX5rtPvJ_5vdYOyGgS5R6Y/edit#gid=312723487
I re-ran one of the queries used there and it looks like we still have the raw data in our analytics database. I'm not completely sure, but it's probably possible to make this data available in some form. I'll have to check with @mpopov about what the procedure would be for that.
@EBernhardson Thanks for pointing to the spreadsheet. It would be really great, if you can make the (anonymized) raw data available so that we can prepare our study.
@mschwarzer can you be more specific what data you need. Maybe it would be sufficient to get the schema information. Thus you can write your queries to test if the data you need is recorded.
Actually this A/B test has many issues that I think warrant holding on making that data available as any kind of example. Namely, we do not even know if the variable we are measuring has a higher intrinsic variation than the one measured in the test. cc @EBernhardson @mpopov
Also, as a side note any data made available in granular form has to be vetted by Security and Legal. Making aggregated results available is OK.
Can you provide the database schema where the data is stored? Then, I can create a query for the aggregation.