When T252685 is done, setup an interleaved A/B test with existing commons search to make sure it isn't any worse
If it isn't any worse, then we can begin tuning it to make it better (if it is we consider what to do next)
Note: after a successful A/B test, revert WikibaseMediaInfoHooks.php so that it only loads the MediaQueryBuilder for NS_FILE
Change 616529 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/WikibaseMediaInfo@master] Always register the media info builder
Change 616530 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] MediaSearch A/B test on commons
Change 616529 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Always register the media info builder
Change 616559 had a related patch set uploaded (by Cparle; owner: Cparle):
[mediawiki/extensions/WikimediaEvents@master] Interleaved A/B test for searching using new commons-specific elasticsearch query builder
We still need to determine how long we're going to run the test for, and how we're going to evaluate the results. Hoping that the Search team has some standards around this that we can use - @dcausse , @EBernhardson?
Also, do we need to let the community know that we're running the A/B test - @Keegan?
We still need to determine how long we're going to run the test for, and how we're going to evaluate the results. Hoping that the Search team has some standards around this that we can use - @dcausse , @EBernhardson?
For test length, typically 7 days but it depends on the amounts of data collected. For many wikis 7 is enough though.
Also, do we need to let the community know that we're running the A/B test - @Keegan?
We don't usually bring up tests to the users, user behaviour is different when they know something is going on which biases the results.
I'm thinking the ideal 7 day period would be August 24 - August 31st - any objections? @nettrom_WMF will be on vacation, but I think we should still be okay to evaluate the search satisfaction score without him. @EBernhardson does that sounds good to you?
Any 7 day period will work, I'll be starting vacation the 31st but I'm sure everything can still be figured out.
Change 616530 merged by jenkins-bot:
[operations/mediawiki-config@master] MediaSearch A/B test on commons
Change 621099 had a related patch set uploaded (by Ebernhardson; owner: Ebernhardson):
[operations/mediawiki-config@master] Correct CirrusSearchUserTesting configuration
Change 621099 merged by jenkins-bot:
[operations/mediawiki-config@master] Correct CirrusSearchUserTesting configuration
Change 621706 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/WikimediaEvents@master] Enable MediaSearch A/B test
Change 621706 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Enable MediaSearch A/B test
Change 621744 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.5] Enable MediaSearch A/B test
Change 621744 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.36.0-wmf.5] Enable MediaSearch A/B test
Mentioned in SAL (#wikimedia-operations) [2020-08-24T11:43:50Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.5/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: 1066ecbe2836e69211c905f597ad6b62241528c0: Enable MediaSearch A/B test (T254388) (duration: 00m 56s)
Change 616559 abandoned by DCausse:
[mediawiki/extensions/WikimediaEvents@master] Interleaved A/B test for searching using new commons-specific elasticsearch query builder
Reason:
superseded by I5fe11e7ad33cde81ac6876f4b5774f52263862b0
This basically broke the search for me, see https://commons.wikimedia.org/wiki/Commons:Village_pump#cirrusUserTesting=mediasearch_commons_breaks_all_my_queries,_how_to_turn_it_off? .
What is wrong with you people? Why do you unleash poorly tested junk on us without announcing it and without offering the option to opt-out?