Create Task
Maniphest T209402

A/B testing plan for wbsearchentities, context=item
Closed, ResolvedPublic
Actions

Description

We have a couple metrics implemented in relforge that can analyze the results:

  • MRR
  • pFound / eFound

These know how to read the WikidataCompletionSearchClicks eventlogging schema, but that schema doesn't have any affordances for AB testing.

  • Add AB test bucket to eventlogging schema
  • Update javascript to sample users into testing buckets.
  • Update relforge to split metrics by test bucket
  • Create backend test profiles

The test should be set up this way:

  • Config value that enables the whole test (in mediawiki-config)
  • Front end decides on enabling the test for particular request and bucketing (so far we enable only for "en" language and items)
  • Front end adds parameters to request to set test profile - cirrusWBProfile and cirrusRescoreProfile.
  • Backend just uses the test profiles (which need to be set up) to deliver results
  • Front end logs the test bucket together with the results in WikidataCompletionSearchClicks

Related Objects
Search...

EBernhardson added a comment.Nov 13 2018, 6:12 PM

For actually putting users into test buckets on the backend it looks like we will only need to modify the profile that comes back for {type=wikibase_prefix_querybuilder, context=wikibase_prefix_search}. Attaching an appropriate overrider should allow swapping out the name of the profile to use for one provided in the query string.

Ideally the frontend test should be triggered by some variables set by the backend. For small config data like this the ResourceLoaderGetConfigVars hook is where that is configured. That allows turning tests on/off with only
a mediawiki-config patch. This is called once every ~5 minutes and the results cached and provided to all browsers when mw is initializing.

Smalyshev triaged this task as Normal priority.Nov 13 2018, 6:48 PM
Smalyshev updated the task description. (Show Details)Nov 13 2018, 7:15 PM
gerritbot added a subscriber: gerritbot.Nov 13 2018, 8:52 PM

Change 473279 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/Wikibase@master] Expose entitysearch api parameters to hook

https://gerrit.wikimedia.org/r/473279

gerritbot added a comment.Nov 13 2018, 9:42 PM

Change 473284 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/473284

gerritbot added a comment.Nov 15 2018, 7:20 AM

Change 473279 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Expose entitysearch api parameters to hook

https://gerrit.wikimedia.org/r/473279

gerritbot added a comment.Nov 20 2018, 4:30 PM

Change 474947 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/Wikibase@master] wbsearchentities: tie-breaker and title-exact profile options

https://gerrit.wikimedia.org/r/474947

EBernhardson added a comment.EditedNov 26 2018, 6:29 PM

Analysis of tuning process: F27316679

For the ab test it looks like the variables are split into three places. The values chosen below are mostly chopped to a single decimal place, as the individual variables aren't that sensitive. Only tie-breaker was given an extra decimal to 0.15. Loading these reduced precision values into the model and comparing the 95% CI, this tune and the exact values overlap significantly.

Stored into $wgWBRepoSettings['entitySearch']['prefixSearchProfiles'][name_of_profile]:

Var nameprofile vartuned value
labels.en.near_matchlang-exact0.2
labels.en.near_match_foldedlang-folded0.1
labels.en.prefixlang-prefix0.6
labels_all.near_match_foldedany0.4
tie_breakertie-breaker0.35

Stored into $wgWBRepoSettings['entitySearch']['rescoreProfiles'][name_of_profile]:

Var nameprofile vartuned value
query_weightquery_weight1.0
rescore_query_weightrescore_query_weight0.9

Stored into $wgWBRepoSettings['entitySearch']['rescoreFunctionChains'][name_of_chain]:

Var nameprofile vartuned valuechosen value
satu/incoming_links/afunctions[0].params.a0.3
satu/incoming_links/kfunctions[0].params.k400
satu/incoming_links/weightfunctions[0].weight0.9
satu/sitelink_count/afunctions[1].params.a0.8
satu/sitelink_count/kfunctions[1].params.k80
satu/weightfunctions[1].weight0.15
gerritbot added a comment.Nov 26 2018, 6:50 PM

Change 475817 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Search profiles for wbsearchentities AB test

https://gerrit.wikimedia.org/r/475817

gerritbot added a comment.Nov 26 2018, 7:18 PM

Change 475817 merged by jenkins-bot:
[operations/mediawiki-config@master] Search profiles for wbsearchentities AB test

https://gerrit.wikimedia.org/r/475817

Stashbot added a subscriber: Stashbot.Nov 26 2018, 7:33 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-26T19:33:42Z] <ebernhardson@deploy1001> Synchronized wmf-config/WikibaseSearchSettings.php: SWAT T209402 Search profiles for wbsearchentities AB test (duration: 00m 46s)

gerritbot added a comment.Nov 26 2018, 7:45 PM

Change 475826 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.4] wbsearchentities: Add tie-breaker search profile option

https://gerrit.wikimedia.org/r/475826

gerritbot added a comment.Nov 26 2018, 8:08 PM

Change 475826 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.4] wbsearchentities: Add tie-breaker search profile option

https://gerrit.wikimedia.org/r/475826

Stashbot added a comment.Nov 26 2018, 8:14 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-26T20:14:11Z] <ebernhardson@deploy1001> Synchronized php-1.33.0-wmf.4/extensions/Wikibase/repo/includes/Search/Elastic/EntitySearchElastic.php: SWAT T209402 Make wbsearchentities tie-breaker configurable (duration: 00m 47s)

gerritbot added a comment.Nov 26 2018, 8:20 PM

Change 474947 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] wbsearchentities: Add tie-breaker search profile option

https://gerrit.wikimedia.org/r/474947

gerritbot added a comment.Nov 27 2018, 9:26 PM

Change 476141 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Start wbsearchentities AB test

https://gerrit.wikimedia.org/r/476141

EBernhardson updated the task description. (Show Details)Nov 27 2018, 9:29 PM

This should be all ready to go. Will need to SWAT out the AB testing code from WikimediaEvents along with the testing configuration in mediawiki-config

gerritbot added a comment.Nov 27 2018, 11:12 PM

Change 473284 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/473284

gerritbot added a comment.Nov 27 2018, 11:13 PM

Change 476176 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.4] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/476176

gerritbot added a comment.Nov 27 2018, 11:13 PM

Change 476177 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.6] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/476177

gerritbot added a comment.Nov 28 2018, 12:20 AM

Change 476176 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.4] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/476176

gerritbot added a comment.Nov 28 2018, 12:22 AM

Change 476177 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.6] AB testing of WikidataCompletionSearchClicks

https://gerrit.wikimedia.org/r/476177

gerritbot added a comment.Nov 28 2018, 12:23 AM

Change 476141 merged by jenkins-bot:
[operations/mediawiki-config@master] Start wbsearchentities AB test

https://gerrit.wikimedia.org/r/476141

gerritbot added a comment.Nov 28 2018, 12:35 AM

Change 476182 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.4] Expose entitysearch api parameters to hook

https://gerrit.wikimedia.org/r/476182

gerritbot added a comment.Nov 28 2018, 12:57 AM

Change 476182 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.33.0-wmf.4] Expose entitysearch api parameters to hook

https://gerrit.wikimedia.org/r/476182

Stashbot added a comment.Nov 28 2018, 1:08 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-28T01:08:44Z] <ebernhardson@deploy1001> Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 53s)

Stashbot added a comment.Nov 28 2018, 1:09 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-28T01:09:50Z] <ebernhardson@deploy1001> Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 52s)

Stashbot added a comment.Nov 28 2018, 1:12 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-28T01:12:38Z] <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T209402: Configuration for wbsearchentities AB test (duration: 00m 53s)

gerritbot added a comment.Nov 28 2018, 1:27 AM

Change 476188 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Start wbsearchentities AB test

https://gerrit.wikimedia.org/r/476188

gerritbot added a comment.Nov 28 2018, 1:31 AM

Change 476188 merged by jenkins-bot:
[operations/mediawiki-config@master] Start wbsearchentities AB test

https://gerrit.wikimedia.org/r/476188

Stashbot added a comment.Nov 28 2018, 1:34 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-28T01:34:45Z] <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T209402: Start wbsearchentities ab test at 10% (duration: 00m 54s)

EBernhardson added a comment.Nov 28 2018, 2:40 AM

Test is shipped out. 10% of english item searches will use the newly tuned parameters. Another 10% will use the classic SQL search. Test is intended to run one week, to be disabled on Dec 4.

gerritbot added a comment.Nov 28 2018, 4:27 PM

Change 476303 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/mediawiki-config@master] Disable classic_entity wbsearchentities AB test

https://gerrit.wikimedia.org/r/476303

Ladsgroup added a subscriber: Ladsgroup.Nov 28 2018, 4:55 PM

Hey, I just wanted to say please contact the development team before doing something as major as this. The wb_terms table in production is fragmented (we emptied lots of its data) and never should be used as the search backend, specially in prod. The plan is to completely drop it. It's worth noting that this table has been cause of outages several times before.

gerritbot added a comment.Nov 28 2018, 5:02 PM

Change 476303 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable classic_entity wbsearchentities AB test

https://gerrit.wikimedia.org/r/476303

Stashbot added a comment.Nov 28 2018, 5:11 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-28T17:11:12Z] <thcipriani@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476303|Disable classic_entity wbsearchentities AB test]] T209402 T210618 (duration: 00m 55s)

gerritbot added a comment.Nov 29 2018, 12:28 AM

Change 476433 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@master] wbsearchentities ab test improvements

https://gerrit.wikimedia.org/r/476433

gerritbot added a comment.Nov 29 2018, 10:57 PM

Change 476772 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Update wbsearchentities ab test configuration

https://gerrit.wikimedia.org/r/476772

gerritbot added a comment.Nov 29 2018, 11:21 PM

Change 476433 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] wbsearchentities ab test improvements

https://gerrit.wikimedia.org/r/476433

gerritbot added a comment.Nov 29 2018, 11:40 PM

Change 476777 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.6] wbsearchentities ab test improvements

https://gerrit.wikimedia.org/r/476777

gerritbot added a comment.Nov 30 2018, 12:29 AM

Change 476777 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.33.0-wmf.6] wbsearchentities ab test improvements

https://gerrit.wikimedia.org/r/476777

Stashbot added a comment.Nov 30 2018, 1:06 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-30T01:06:40Z] <thcipriani@deploy1001> Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents: SWAT: [[gerrit:476777|wbsearchentities ab test improvements]] T209402 (duration: 00m 46s)

gerritbot added a comment.Nov 30 2018, 2:13 AM

Change 476772 merged by jenkins-bot:
[operations/mediawiki-config@master] Update wbsearchentities ab test configuration

https://gerrit.wikimedia.org/r/476772

Stashbot added a comment.Nov 30 2018, 2:17 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-30T02:17:04Z] <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:476772|update wbsearchentities ab test configuration]] T209402 (duration: 00m 47s)

gerritbot added a comment.Dec 6 2018, 9:56 PM

Change 478103 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Turn off wbsearchentities test

https://gerrit.wikimedia.org/r/478103

gerritbot added a comment.Dec 7 2018, 12:04 AM

Change 478103 merged by jenkins-bot:
[operations/mediawiki-config@master] Turn off wbsearchentities test

https://gerrit.wikimedia.org/r/478103

Stashbot added a comment.Dec 7 2018, 12:10 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-07T00:10:14Z] <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:478103|turn off wbsearchentities ab test]] T209402 (duration: 00m 47s)

debt closed this task as Resolved.Dec 10 2018, 9:18 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL