Page MenuHomePhabricator

Add skin version and search version fields to search satisfaction schema
Closed, ResolvedPublic5 Estimated Story Points

Description

Background

We will be reusing the search satisfaction schema to perform two A/B tests on our changes to search:

  1. The first on the new location of the search widget.
  2. The second on the new widget / experience we're currently building in Vue.js.

These two A/B tests together support four possible configurations:

skinVersion inputLocationextraParamsDescription
legacy"header-navigation"Vector Legacy skin with Legacy search (current master)
latest"header-navigation"Vector Latest skin with Legacy search (current master)
latest"header-moved"Vector Latest skin with Legacy search and latest location
latest"header-moved""WVUI"Vector Latest skin with Latest search and latest location

Both A/B tests will be executed in Vector's "Latest" (v2) mode only and compared to the Legacy mode (v1). In order to do this, we need to be able to distinguish between the old and new versions of the skin. As such, this task requires that a new skin version field be added. This field will only be needed for the search project and could be deleted later.

For the second test only, search can be configured within Vector's Latest mode to use the new experience (Vue.js) or the old experience. This will be set by wiki for at least logged-in users, although logged-in + anon is preferred. These will be distinguished in instrumentation by the addition of a second field for search version. We will also monitor the opt-in / out rate when new search is enabled and disabled.

Acceptance criteria

Once the schema is updated (bumping it to 1.2.0, most likely), the version will also need to be updated in extension.json in WikimediaEvents:

"attributes": {
		"EventLogging": {
			"Schemas": {
				"SearchSatisfaction": "/analytics/legacy/searchsatisfaction/1.1.0",

Open/example patches

Developer notes

  • Ensure that our changes are isolated and can be disabled/reverted easily
  • For completeness, Vector\Hooks::onOutputPageBodyAttributes should be updated to add a CSS class when $wgVectorUseCoreSearch is truthy. Otherwise we'll have to infer the presence of the Vue.js search widget implementation via something like
const isVueJSSearchWidgetPresent = !!document.querySelector( '#app .wvui-input' );

QA Results - Prod

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Niedzielski renamed this task from Add skin version field to search satisfaction schema to Add skin version and search version fields to search satisfaction schema.Aug 3 2020, 5:58 PM
Niedzielski updated the task description. (Show Details)
Niedzielski updated the task description. (Show Details)

I've updated the task description with my understanding from standup. Please review and amend as needed, @ovasileva, then unassign yourself /cc @phuedx, our instrumentation implementation master

Since we can only report skin with the current schema + skin version, how can we determine the configuration? I think we need to either correlate a report with a deployment time or add a second variable.

Agreed. The addition of the search version (search treatment?) to the table in the description makes sense.

mpopov updated the task description. (Show Details)Aug 4 2020, 3:14 PM
mpopov updated the task description. (Show Details)
mpopov added a comment.EditedAug 4 2020, 3:17 PM

I've updated the task description, since SearchSatisfaction has been migrated from legacy EventLogging to Event Platform.

phuedx claimed this task.Aug 12 2020, 2:05 PM

Change 620010 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] [Hooks][PHP] Don't use SkinVersionLookup directly

https://gerrit.wikimedia.org/r/620010

Change 620011 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] [Hooks][PHP] Document CSS classes used elsewhere

https://gerrit.wikimedia.org/r/620011

Change 620012 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/extensions/WikimediaEvents@master] SearchSatisfaction: Add skinVersion and searchVersion fields

https://gerrit.wikimedia.org/r/620012

mpopov updated the task description. (Show Details)Aug 13 2020, 3:30 PM

Change 620011 merged by jenkins-bot:
[mediawiki/skins/Vector@master] hooks: Document CSS classes used elsewhere

https://gerrit.wikimedia.org/r/620011

Change 620010 merged by jenkins-bot:
[mediawiki/skins/Vector@master] hooks: Don't use SkinVersionLookup directly

https://gerrit.wikimedia.org/r/620010

Change 621473 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[schemas/event/secondary@master] SearchSatisfaction: Add skin, skinVersion, and searchTreatment fields

https://gerrit.wikimedia.org/r/621473

phuedx updated the task description. (Show Details)Aug 25 2020, 11:12 AM
phuedx removed phuedx as the assignee of this task.Aug 25 2020, 11:58 AM
phuedx updated the task description. (Show Details)
ovasileva set the point value for this task to 5.Aug 25 2020, 4:31 PM

Change 621473 merged by Bearloga:
[schemas/event/secondary@master] SearchSatisfaction: Add skin and skinVersion fields

https://gerrit.wikimedia.org/r/621473

phuedx added a comment.Sep 9 2020, 4:22 PM

Thanks for the ping, @Jdlrobson! I've updated the patch and removed my -2.

Change 620012 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] SearchSatisfaction: Set fields for search widget A/B tests in Vector

https://gerrit.wikimedia.org/r/620012

Jdlrobson assigned this task to phuedx.Sep 9 2020, 6:44 PM

Might need QA but not sure what that QA should be. Leaving with Sam to determine.

phuedx added a subscriber: Edtadros.

This is particularly tricky to test anywhere other that your local development environment as the sampling/bucketing code that enables the instrument is in WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js. That being said, it doesn't mean it shouldn't be QA'd. I'd be happy to pair with @Edtadros on this.

phuedx reassigned this task from phuedx to Edtadros.Sep 15 2020, 3:06 PM
phuedx claimed this task.Sep 15 2020, 3:42 PM

Per the SearchSatisfaction Instrumentation & Data-QA meeting, I'll add more notes about fields that will be populated and for @Edtadros to do QA in his local development environment.

Change 627819 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/core@master] skins: Increase visibility of SkinMustache::buildSearchProps

https://gerrit.wikimedia.org/r/627819

Change 627820 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] SearchBox: Add data-search-loc attribute

https://gerrit.wikimedia.org/r/627820

phuedx added a comment.EditedSep 16 2020, 3:47 PM

A couple of notes about the changes that've been made and what to expect:

  • The skin and skinVersion properties have been added to the SearchSatisfaction and the schema's minor version number has been bumped
    • The schema's version is 1.2.0
  • The SearchSatisfaction instrument sets the skin property for _every_ event
  • If the user is using the Vector skin, the SearchSatisfaction schema:
    • Sets the skinVersion property to "legacy" or "latest"
    • Appends "WVUI" to the extraParams property, if the user has been sent the Vue.js-based search widget
    • Once https://gerrit.wikimedia.org/r/627820 is merged, the inputLocation property will be set to either "header-navigation" or "header-moved"

The description of this task mentions adding a searchVersion property to the schema. However, as you can see above, the schema already accounted for both the location of the search widget and additional metadata to be added. We can still compute the property in our queries though:

select
  case
    when event.skinVersion = 'latest' and event.inputLocation = 'header-moved' and instr(event.extraParams, 'WVUI') > 0 then 'new'
    when event.skinVersion = 'latest' and event.inputLocation = 'header-moved' then 'moved'
    else 'legacy'
  end as skinVersion
from
  event.SchemaSatisfaction
where
  event.skin = 'vector'
# ...
;
MNeisler updated the task description. (Show Details)Sep 16 2020, 4:51 PM
ovasileva reassigned this task from phuedx to Edtadros.Sep 16 2020, 5:07 PM
phuedx removed Edtadros as the assignee of this task.Sep 16 2020, 5:09 PM

Change 627819 abandoned by Jdlrobson:
[mediawiki/core@master] skins: Increase visibility of SkinMustache::buildSearchProps

Reason:
I don't think this is needed now that https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/ /627820 has been 2ed but there may be follow up work to do here e.g. mark methods as final. I've started a private message channel with Sam to work out next steps.

https://gerrit.wikimedia.org/r/627819

Change 627820 merged by jenkins-bot:
[mediawiki/skins/Vector@master] SearchBox: Add data-search-loc attribute

https://gerrit.wikimedia.org/r/627820

This will be QA'd alongside T259250.

phuedx added a comment.EditedSep 22 2020, 12:33 PM

Here's my notes on setting up my local development environment (MediaWiki Docker) to test this:

  • Load the WikimediaEvents extension
    • Load the EventLogging and EventStreamsConfig extensions
    • Add $wgEventLoggingBaseUri = $wgServer . '/beacon/event'; to LocalSettings.php
  • Enable EventLogging's debug mode by running the following in my browser console:
new mw.Api().saveOption( 'eventlogging-display-web', '1' );
  • Increase the SearchSatisfaction instrumentation's sample size to 100%, i.e.
WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js
// Line 132
sampleSize = {
  // % of sessions to sample
  test: 100
  // ...
},

Change 629675 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] SearchBox: Fix data-search-loc attribute

https://gerrit.wikimedia.org/r/629675

phuedx removed Edtadros as the assignee of this task.Sep 24 2020, 12:52 PM

Change 629675 merged by jenkins-bot:
[mediawiki/skins/Vector@master] SearchBox: Fix data-search-loc attribute

https://gerrit.wikimedia.org/r/629675

Change 630418 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@wmf/1.36.0-wmf.10] SearchBox: Fix data-search-loc attribute

https://gerrit.wikimedia.org/r/630418

@Jdlrobson: There's the backport of the fix that's a deployment blocker.

Change 630418 abandoned by Jdlrobson:
[mediawiki/skins/Vector@wmf/1.36.0-wmf.10] SearchBox: Fix data-search-loc attribute

Reason:
Talked to Sam and this doesn't need to be backported - a/b test only applies to modern Vector

https://gerrit.wikimedia.org/r/630418

Edtadros added a comment.EditedThu, Oct 8, 1:36 PM

Test Result - Prod

Status: ✅ PASS
Environment: hewiki
OS: macOS Catalina
Browser: Chrome
Device: MBP
Emulated Device: NA

Test Artifact(s):

QA Steps

Verify the event elements below:

ACskinVersion Expected inputLocationextraParamsEvent
✅ AC1legacy"header-navigation"Vector Legacy skin with Legacy search (current master)
action: "searchResultPage"
autocompleteType: "comp_suggest"
hitsReturned: 10
inputLocation: "header-navigation"
msToDisplayResults: 181
mwSessionId: "98a750337dc29632f333"
pageViewId: "5b2d44f2432a14418c0fkg5u4iud"
query: "d"
sampleMultiplier: 1
scroll: false
searchSessionId: "a23c5846edd26107b9d3kg5u4iud"
searchToken: "akgvwvuqg40khwgyzx7lri912"
skin: "vector"
skinVersion: "legacy"
source: "autocomplete"
uniqueId: "cda9f04112ed3644f9a0kg5u4j30"
✅ AC2latest"header-navigation"Vector Latest skin with Legacy search (current master)
action: "searchResultPage"
autocompleteType: "comp_suggest"
hitsReturned: 10
inputLocation: "header-navigation"
msToDisplayResults: 199
mwSessionId: "98a750337dc29632f333"
pageViewId: "e2e17ff5412567c697f4kg5u8s23"
query: "d"
sampleMultiplier: 1
scroll: false
searchSessionId: "a23c5846edd26107b9d3kg5u4iud"
searchToken: "5pbtk3lznf26dhbwpk7ckbwme"
skin: "vector"
skinVersion: "latest"
source: "autocomplete"
uniqueId: "89027b22453e50db6253kg5u8sb1"
✅ AC3latest"header-moved"Vector Latest skin with Legacy search and latest location
action: "searchResultPage"
autocompleteType: "comp_suggest"
hitsReturned: 10
inputLocation: "header-moved"
msToDisplayResults: 142
mwSessionId: "98a750337dc29632f333"
pageViewId: "7b75032b6e60b64cb8c0kg5ucu1h"
query: "d"
sampleMultiplier: 1
scroll: false
searchSessionId: "a23c5846edd26107b9d3kg5u4iud"
searchToken: "7stm87kmx4om2sy6a0mcvtcmm"
skin: "vector"
skinVersion: "latest"
source: "autocomplete"
uniqueId: "7270da579ba087b8c3e8kg5ucu8v"
✅ AC4latest"header-moved""WVUI"Vector Latest skin with Latest search and latest location
event: {action: "searchResultPage", source: "autocomplete", searchSessionId: "a59f37df9d98ac074ecdkg0uvicj",…}
action: "searchResultPage"
autocompleteType: "comp_suggest"
hitsReturned: 10
inputLocation: "header-moved"
msToDisplayResults: 192
mwSessionId: "98a750337dc29632f333"
pageViewId: "6b41f5080f145b18d34bkg0v34id"
query: "ge"
sampleMultiplier: 1
scroll: false
searchSessionId: "a59f37df9d98ac074ecdkg0uvicj"
searchToken: "c5lb015hlrn6xm2oyvdal48mg"
skin: "vector"
skinVersion: "latest"
source: "autocomplete"
uniqueId: "6a53debaa3e7fd798f6bkg0v34uk"
Edtadros updated the task description. (Show Details)Thu, Oct 8, 1:38 PM
Edtadros reassigned this task from Edtadros to ovasileva.Mon, Oct 12, 1:23 AM
Edtadros updated the task description. (Show Details)
ovasileva closed this task as Resolved.Tue, Oct 13, 11:08 AM

All done!