I've updated the task description with my understanding from standup. Please review and amend as needed, @ovasileva, then unassign yourself /cc @phuedx, our instrumentation implementation master

In T256100#6356174, @Niedzielski wrote:

Since we can only report skin with the current schema + skin version, how can we determine the configuration? I think we need to either correlate a report with a deployment time or add a second variable.

Agreed. The addition of the search version (search treatment?) to the table in the description makes sense.

I've updated the task description, since SearchSatisfaction has been migrated from legacy EventLogging to Event Platform.

Change 620010 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] [Hooks][PHP] Don't use SkinVersionLookup directly

https://gerrit.wikimedia.org/r/620010

Change 620011 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] [Hooks][PHP] Document CSS classes used elsewhere

https://gerrit.wikimedia.org/r/620011

Change 620012 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/extensions/WikimediaEvents@master] SearchSatisfaction: Add skinVersion and searchVersion fields

https://gerrit.wikimedia.org/r/620012

Change 620011 merged by jenkins-bot:
[mediawiki/skins/Vector@master] hooks: Document CSS classes used elsewhere

https://gerrit.wikimedia.org/r/620011

Change 620010 merged by jenkins-bot:
[mediawiki/skins/Vector@master] hooks: Don't use SkinVersionLookup directly

https://gerrit.wikimedia.org/r/620010

Change 621473 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[schemas/event/secondary@master] SearchSatisfaction: Add skin, skinVersion, and searchTreatment fields

https://gerrit.wikimedia.org/r/621473

Change 621473 merged by Bearloga:
[schemas/event/secondary@master] SearchSatisfaction: Add skin and skinVersion fields

https://gerrit.wikimedia.org/r/621473

All patches except https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/620012 are merged which is -2ed.

Thanks for the ping, @Jdlrobson! I've updated the patch and removed my -2.

Change 620012 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] SearchSatisfaction: Set fields for search widget A/B tests in Vector

https://gerrit.wikimedia.org/r/620012

This is particularly tricky to test anywhere other that your local development environment as the sampling/bucketing code that enables the instrument is in WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js. That being said, it doesn't mean it shouldn't be QA'd. I'd be happy to pair with @Edtadros on this.

Change 627819 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/core@master] skins: Increase visibility of SkinMustache::buildSearchProps

https://gerrit.wikimedia.org/r/627819

Change 627820 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] SearchBox: Add data-search-loc attribute

https://gerrit.wikimedia.org/r/627820

A couple of notes about the changes that've been made and what to expect:

• The skin and skinVersion properties have been added to the SearchSatisfaction and the schema's minor version number has been bumped
  • The schema's version is 1.2.0
• The SearchSatisfaction instrument sets the skin property for _every_ event
• If the user is using the Vector skin, the SearchSatisfaction schema:
  • Sets the skinVersion property to "legacy" or "latest"
  • Appends "WVUI" to the extraParams property, if the user has been sent the Vue.js-based search widget
  • Once https://gerrit.wikimedia.org/r/627820 is merged, the inputLocation property will be set to either "header-navigation" or "header-moved"

The description of this task mentions adding a searchVersion property to the schema. However, as you can see above, the schema already accounted for both the location of the search widget and additional metadata to be added. We can still compute the property in our queries though:

select
  case
    when event.skinVersion = 'latest' and event.inputLocation = 'header-moved' and instr(event.extraParams, 'WVUI') > 0 then 'new'
    when event.skinVersion = 'latest' and event.inputLocation = 'header-moved' then 'moved'
    else 'legacy'
  end as skinVersion
from
  event.SchemaSatisfaction
where
  event.skin = 'vector'
# ...
;

Change 627819 abandoned by Jdlrobson:
[mediawiki/core@master] skins: Increase visibility of SkinMustache::buildSearchProps

Reason:
I don't think this is needed now that https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/ /627820 has been 2ed but there may be follow up work to do here e.g. mark methods as final. I've started a private message channel with Sam to work out next steps.

https://gerrit.wikimedia.org/r/627819

Change 627820 merged by jenkins-bot:
[mediawiki/skins/Vector@master] SearchBox: Add data-search-loc attribute

https://gerrit.wikimedia.org/r/627820

This will be QA'd alongside T259250.

Here's my notes on setting up my local development environment (MediaWiki Docker) to test this:

• Load the WikimediaEvents extension
  • Load the EventLogging and EventStreamsConfig extensions
  • Add $wgEventLoggingBaseUri = $wgServer . '/beacon/event'; to LocalSettings.php

• Enable EventLogging's debug mode by running the following in my browser console:

new mw.Api().saveOption( 'eventlogging-display-web', '1' );

• Increase the SearchSatisfaction instrumentation's sample size to 100%, i.e.

// Line 132
sampleSize = {
  // % of sessions to sample
  test: 100
  // ...
},

Change 629675 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@master] SearchBox: Fix data-search-loc attribute

https://gerrit.wikimedia.org/r/629675

Change 629675 merged by jenkins-bot:
[mediawiki/skins/Vector@master] SearchBox: Fix data-search-loc attribute

https://gerrit.wikimedia.org/r/629675

Change 630418 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/skins/Vector@wmf/1.36.0-wmf.10] SearchBox: Fix data-search-loc attribute

https://gerrit.wikimedia.org/r/630418

@Jdlrobson: There's the backport of the fix that's a deployment blocker.

Change 630418 abandoned by Jdlrobson:
[mediawiki/skins/Vector@wmf/1.36.0-wmf.10] SearchBox: Fix data-search-loc attribute

Reason:
Talked to Sam and this doesn't need to be backported - a/b test only applies to modern Vector

https://gerrit.wikimedia.org/r/630418

Test Result - Prod

Status: ✅ PASS
Environment: hewiki
OS: macOS Catalina
Browser: Chrome
Device: MBP
Emulated Device: NA

Test Artifact(s):

QA Steps

Verify the event elements below:

action: "searchResultPage"
autocompleteType: "comp_suggest"
hitsReturned: 10
inputLocation: "header-navigation"
msToDisplayResults: 181
mwSessionId: "98a750337dc29632f333"
pageViewId: "5b2d44f2432a14418c0fkg5u4iud"
query: "d"
sampleMultiplier: 1
scroll: false
searchSessionId: "a23c5846edd26107b9d3kg5u4iud"
searchToken: "akgvwvuqg40khwgyzx7lri912"
skin: "vector"
skinVersion: "legacy"
source: "autocomplete"
uniqueId: "cda9f04112ed3644f9a0kg5u4j30"

action: "searchResultPage"
autocompleteType: "comp_suggest"
hitsReturned: 10
inputLocation: "header-navigation"
msToDisplayResults: 199
mwSessionId: "98a750337dc29632f333"
pageViewId: "e2e17ff5412567c697f4kg5u8s23"
query: "d"
sampleMultiplier: 1
scroll: false
searchSessionId: "a23c5846edd26107b9d3kg5u4iud"
searchToken: "5pbtk3lznf26dhbwpk7ckbwme"
skin: "vector"
skinVersion: "latest"
source: "autocomplete"
uniqueId: "89027b22453e50db6253kg5u8sb1"

action: "searchResultPage"
autocompleteType: "comp_suggest"
hitsReturned: 10
inputLocation: "header-moved"
msToDisplayResults: 142
mwSessionId: "98a750337dc29632f333"
pageViewId: "7b75032b6e60b64cb8c0kg5ucu1h"
query: "d"
sampleMultiplier: 1
scroll: false
searchSessionId: "a23c5846edd26107b9d3kg5u4iud"
searchToken: "7stm87kmx4om2sy6a0mcvtcmm"
skin: "vector"
skinVersion: "latest"
source: "autocomplete"
uniqueId: "7270da579ba087b8c3e8kg5ucu8v"

event: {action: "searchResultPage", source: "autocomplete", searchSessionId: "a59f37df9d98ac074ecdkg0uvicj",…}
action: "searchResultPage"
autocompleteType: "comp_suggest"
hitsReturned: 10
inputLocation: "header-moved"
msToDisplayResults: 192
mwSessionId: "98a750337dc29632f333"
pageViewId: "6b41f5080f145b18d34bkg0v34id"
query: "ge"
sampleMultiplier: 1
scroll: false
searchSessionId: "a59f37df9d98ac074ecdkg0uvicj"
searchToken: "c5lb015hlrn6xm2oyvdal48mg"
skin: "vector"
skinVersion: "latest"
source: "autocomplete"
uniqueId: "6a53debaa3e7fd798f6bkg0v34uk"