Page MenuHomePhabricator

Create interleaved A/B test for searching using new commons-specific elasticsearch query builder
Closed, ResolvedPublic

Description

When T252685 is done, setup an interleaved A/B test with existing commons search to make sure it isn't any worse

If it isn't any worse, then we can begin tuning it to make it better (if it is we consider what to do next)

Note: after a successful A/B test, revert WikibaseMediaInfoHooks.php so that it only loads the MediaQueryBuilder for NS_FILE

Event Timeline

Cparle created this task.Jun 3 2020, 7:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 3 2020, 7:00 PM
Cparle updated the task description. (Show Details)Jul 17 2020, 11:22 AM
Cparle claimed this task.Jul 22 2020, 4:50 PM
Cparle moved this task from Incoming to Doing on the Structured-Data-Backlog (Current Work) board.

Change 616529 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/WikibaseMediaInfo@master] Always register the media info builder

https://gerrit.wikimedia.org/r/616529

Change 616530 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] MediaSearch A/B test on commons

https://gerrit.wikimedia.org/r/616530

Change 616529 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Always register the media info builder

https://gerrit.wikimedia.org/r/616529

Change 616559 had a related patch set uploaded (by Cparle; owner: Cparle):
[mediawiki/extensions/WikimediaEvents@master] Interleaved A/B test for searching using new commons-specific elasticsearch query builder

https://gerrit.wikimedia.org/r/616559

We still need to determine how long we're going to run the test for, and how we're going to evaluate the results. Hoping that the Search team has some standards around this that we can use - @dcausse , @EBernhardson?

Also, do we need to let the community know that we're running the A/B test - @Keegan?

We still need to determine how long we're going to run the test for, and how we're going to evaluate the results. Hoping that the Search team has some standards around this that we can use - @dcausse , @EBernhardson?

For test length, typically 7 days but it depends on the amounts of data collected. For many wikis 7 is enough though.

Also, do we need to let the community know that we're running the A/B test - @Keegan?

We don't usually bring up tests to the users, user behaviour is different when they know something is going on which biases the results.

I'm thinking the ideal 7 day period would be August 24 - August 31st - any objections? @nettrom_WMF will be on vacation, but I think we should still be okay to evaluate the search satisfaction score without him. @EBernhardson does that sounds good to you?

Any 7 day period will work, I'll be starting vacation the 31st but I'm sure everything can still be figured out.

Change 616530 merged by jenkins-bot:
[operations/mediawiki-config@master] MediaSearch A/B test on commons

https://gerrit.wikimedia.org/r/616530

Change 621099 had a related patch set uploaded (by Ebernhardson; owner: Ebernhardson):
[operations/mediawiki-config@master] Correct CirrusSearchUserTesting configuration

https://gerrit.wikimedia.org/r/621099

Change 621099 merged by jenkins-bot:
[operations/mediawiki-config@master] Correct CirrusSearchUserTesting configuration

https://gerrit.wikimedia.org/r/621099

Change 621706 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/WikimediaEvents@master] Enable MediaSearch A/B test

https://gerrit.wikimedia.org/r/621706

Change 621706 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Enable MediaSearch A/B test

https://gerrit.wikimedia.org/r/621706

Change 621744 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.5] Enable MediaSearch A/B test

https://gerrit.wikimedia.org/r/621744

Change 621744 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.5] Enable MediaSearch A/B test

https://gerrit.wikimedia.org/r/621744

Mentioned in SAL (#wikimedia-operations) [2020-08-24T11:43:50Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: 1066ecbe2836e69211c905f597ad6b62241528c0: Enable MediaSearch A/B test (T254388) (duration: 00m 56s)

Change 616559 abandoned by DCausse:
[mediawiki/extensions/WikimediaEvents@master] Interleaved A/B test for searching using new commons-specific elasticsearch query builder

Reason:
superseded by I5fe11e7ad33cde81ac6876f4b5774f52263862b0

https://gerrit.wikimedia.org/r/616559

This basically broke the search for me, see https://commons.wikimedia.org/wiki/Commons:Village_pump#cirrusUserTesting=mediasearch_commons_breaks_all_my_queries,_how_to_turn_it_off? .
What is wrong with you people? Why do you unleash poorly tested junk on us without announcing it and without offering the option to opt-out?

Gehel closed this task as Resolved.Sep 1 2020, 12:29 PM