A/B testing plan for wbsearchentities, context=item
Closed, ResolvedPublic

Description

We have a couple metrics implemented in relforge that can analyze the results:

  • MRR
  • pFound / eFound

These know how to read the WikidataCompletionSearchClicks eventlogging schema, but that schema doesn't have any affordances for AB testing.

  • Add AB test bucket to eventlogging schema
  • Update javascript to sample users into testing buckets.
  • Update relforge to split metrics by test bucket
  • Create backend test profiles

The test should be set up this way:

  • Config value that enables the whole test (in mediawiki-config)
  • Front end decides on enabling the test for particular request and bucketing (so far we enable only for "en" language and items)
  • Front end adds parameters to request to set test profile - cirrusWBProfile and cirrusRescoreProfile.
  • Backend just uses the test profiles (which need to be set up) to deliver results
  • Front end logs the test bucket together with the results in WikidataCompletionSearchClicks

For actually putting users into test buckets on the backend it looks like we will only need to modify the profile that comes back for {type=wikibase_prefix_querybuilder, context=wikibase_prefix_search}. Attaching an appropriate overrider should allow swapping out the name of the profile to use for one provided in the query string.

Ideally the frontend test should be triggered by some variables set by the backend. For small config data like this the ResourceLoaderGetConfigVars hook is where that is configured. That allows turning tests on/off with only
a mediawiki-config patch. This is called once every ~5 minutes and the results cached and provided to all browsers when mw is initializing.

Smalyshev triaged this task as Normal priority.Tue, Nov 13, 6:48 PM
Smalyshev updated the task description. (Show Details)Tue, Nov 13, 7:15 PM

Change 473279 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/Wikibase@master] Expose entitysearch api parameters to hook

https://gerrit.wikimedia.org/r/473279

Change 473284 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/473284

Change 473279 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Expose entitysearch api parameters to hook

https://gerrit.wikimedia.org/r/473279

Change 474947 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/Wikibase@master] wbsearchentities: tie-breaker and title-exact profile options

https://gerrit.wikimedia.org/r/474947

EBernhardson added a comment.EditedMon, Nov 26, 6:29 PM

Analysis of tuning process: F27316679

For the ab test it looks like the variables are split into three places. The values chosen below are mostly chopped to a single decimal place, as the individual variables aren't that sensitive. Only tie-breaker was given an extra decimal to 0.15. Loading these reduced precision values into the model and comparing the 95% CI, this tune and the exact values overlap significantly.

Stored into $wgWBRepoSettings['entitySearch']['prefixSearchProfiles'][name_of_profile]:

Var nameprofile vartuned value
labels.en.near_matchlang-exact0.2
labels.en.near_match_foldedlang-folded0.1
labels.en.prefixlang-prefix0.6
labels_all.near_match_foldedany0.4
tie_breakertie-breaker0.35

Stored into $wgWBRepoSettings['entitySearch']['rescoreProfiles'][name_of_profile]:

Var nameprofile vartuned value
query_weightquery_weight1.0
rescore_query_weightrescore_query_weight0.9

Stored into $wgWBRepoSettings['entitySearch']['rescoreFunctionChains'][name_of_chain]:

Var nameprofile vartuned valuechosen value
satu/incoming_links/afunctions[0].params.a0.3
satu/incoming_links/kfunctions[0].params.k400
satu/incoming_links/weightfunctions[0].weight0.9
satu/sitelink_count/afunctions[1].params.a0.8
satu/sitelink_count/kfunctions[1].params.k80
satu/weightfunctions[1].weight0.15

Change 475817 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Search profiles for wbsearchentities AB test

https://gerrit.wikimedia.org/r/475817

Change 475817 merged by jenkins-bot:
[operations/mediawiki-config@master] Search profiles for wbsearchentities AB test

https://gerrit.wikimedia.org/r/475817

Mentioned in SAL (#wikimedia-operations) [2018-11-26T19:33:42Z] <ebernhardson@deploy1001> Synchronized wmf-config/WikibaseSearchSettings.php: SWAT T209402 Search profiles for wbsearchentities AB test (duration: 00m 46s)

Change 475826 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.4] wbsearchentities: Add tie-breaker search profile option

https://gerrit.wikimedia.org/r/475826

Change 475826 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.4] wbsearchentities: Add tie-breaker search profile option

https://gerrit.wikimedia.org/r/475826

Mentioned in SAL (#wikimedia-operations) [2018-11-26T20:14:11Z] <ebernhardson@deploy1001> Synchronized php-1.33.0-wmf.4/extensions/Wikibase/repo/includes/Search/Elastic/EntitySearchElastic.php: SWAT T209402 Make wbsearchentities tie-breaker configurable (duration: 00m 47s)

Change 474947 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] wbsearchentities: Add tie-breaker search profile option

https://gerrit.wikimedia.org/r/474947

Change 476141 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Start wbsearchentities AB test

https://gerrit.wikimedia.org/r/476141

EBernhardson updated the task description. (Show Details)Tue, Nov 27, 9:29 PM

This should be all ready to go. Will need to SWAT out the AB testing code from WikimediaEvents along with the testing configuration in mediawiki-config

Change 473284 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/473284

Change 476176 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.4] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/476176

Change 476177 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.6] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/476177

Change 476176 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.4] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/476176

Change 476177 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.6] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/476177

Change 476141 merged by jenkins-bot:
[operations/mediawiki-config@master] Start wbsearchentities AB test

https://gerrit.wikimedia.org/r/476141

Change 476182 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.4] Expose entitysearch api parameters to hook

https://gerrit.wikimedia.org/r/476182

Change 476182 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.4] Expose entitysearch api parameters to hook

https://gerrit.wikimedia.org/r/476182

Mentioned in SAL (#wikimedia-operations) [2018-11-28T01:08:44Z] <ebernhardson@deploy1001> Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2018-11-28T01:09:50Z] <ebernhardson@deploy1001> Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 52s)

Mentioned in SAL (#wikimedia-operations) [2018-11-28T01:12:38Z] <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T209402: Configuration for wbsearchentities AB test (duration: 00m 53s)

Change 476188 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Start wbsearchentities AB test

https://gerrit.wikimedia.org/r/476188

Change 476188 merged by jenkins-bot:
[operations/mediawiki-config@master] Start wbsearchentities AB test

https://gerrit.wikimedia.org/r/476188

Mentioned in SAL (#wikimedia-operations) [2018-11-28T01:34:45Z] <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T209402: Start wbsearchentities ab test at 10% (duration: 00m 54s)

Test is shipped out. 10% of english item searches will use the newly tuned parameters. Another 10% will use the classic SQL search. Test is intended to run one week, to be disabled on Dec 4.

Change 476303 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Disable classic_entity wbsearchentities AB test

https://gerrit.wikimedia.org/r/476303

Hey, I just wanted to say please contact the development team before doing something as major as this. The wb_terms table in production is fragmented (we emptied lots of its data) and never should be used as the search backend, specially in prod. The plan is to completely drop it. It's worth noting that this table has been cause of outages several times before.

Change 476303 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable classic_entity wbsearchentities AB test

https://gerrit.wikimedia.org/r/476303

Mentioned in SAL (#wikimedia-operations) [2018-11-28T17:11:12Z] <thcipriani@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476303|Disable classic_entity wbsearchentities AB test]] T209402 T210618 (duration: 00m 55s)

Change 476433 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] wbsearchentities ab test improvements

https://gerrit.wikimedia.org/r/476433

Change 476772 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Update wbsearchentities ab test configuration

https://gerrit.wikimedia.org/r/476772

Change 476433 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] wbsearchentities ab test improvements

https://gerrit.wikimedia.org/r/476433

Change 476777 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.6] wbsearchentities ab test improvements

https://gerrit.wikimedia.org/r/476777

Change 476777 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.6] wbsearchentities ab test improvements

https://gerrit.wikimedia.org/r/476777

Mentioned in SAL (#wikimedia-operations) [2018-11-30T01:06:40Z] <thcipriani@deploy1001> Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents: SWAT: [[gerrit:476777|wbsearchentities ab test improvements]] T209402 (duration: 00m 46s)

Change 476772 merged by jenkins-bot:
[operations/mediawiki-config@master] Update wbsearchentities ab test configuration

https://gerrit.wikimedia.org/r/476772

Mentioned in SAL (#wikimedia-operations) [2018-11-30T02:17:04Z] <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:476772|update wbsearchentities ab test configuration]] T209402 (duration: 00m 47s)

Change 478103 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Turn off wbsearchentities test

https://gerrit.wikimedia.org/r/478103

Change 478103 merged by jenkins-bot:
[operations/mediawiki-config@master] Turn off wbsearchentities test

https://gerrit.wikimedia.org/r/478103

Mentioned in SAL (#wikimedia-operations) [2018-12-07T00:10:14Z] <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:478103|turn off wbsearchentities ab test]] T209402 (duration: 00m 47s)

debt closed this task as Resolved.Mon, Dec 10, 9:18 PM