Do we need a skin version parameter in the schema?

In T250282#6189608, @Niedzielski wrote:

Do we need a skin version parameter in the schema?

The measurement questions in T245735 does not require us to look at skin version exclusively. Also, the only sessions that would have "collapsed" state would be the ones where users have the new Vector skin. Hence I do not see a need for skin version.
I can check with @ovasileva and @MNeisler to confirm.

In T250282#6190669, @Mayakp.wiki wrote:

In T250282#6189608, @Niedzielski wrote:

Do we need a skin version parameter in the schema?

The measurement questions in T245735 does not require us to look at skin version exclusively. Also, the only sessions that would have "collapsed" state would be the ones where users have the new Vector skin. Hence I do not see a need for skin version.
I can check with @ovasileva and @MNeisler to confirm.

So originally we wanted this to establish a baseline for the donate button, thus we'd want the instrumentation to work on both the old and new versions of vector. That said, we can get this info by deployment date or using skin version...

Change 604486 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@master] Use packageFiles in ext.wikimediaEvents

https://gerrit.wikimedia.org/r/604486

Change 604487 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@master] Edit count bucket pulled out of mobileWebUIActions schema

https://gerrit.wikimedia.org/r/604487

Change 604492 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@master] WIP: Sidebar instrumentation

https://gerrit.wikimedia.org/r/604492

Discussed in our 1:1 today 06-11-20

In T250282#6189608, @Niedzielski wrote:

Do we need a skin version parameter in the schema?

Decision: we need the instrumentation to run on both versions of the Vector skin hence it makes sense to have skin version added to the schema (to establish a baseline and then be able to compare with users that opt-out later on).

@Jdlrobson brought up 2 questions in the Instrumentation doc:

1. On 'click-donate' action

I was wondering if "click-donate" is needed. As I implement this, I'm seeing complexity with this requirement. Donate events can be uniquely identified by searching for action=click AND name='donate' is that enough ?

• Ok to remove "click-donate" action by searching for action=click AND name='donate'

2. currentState and newState columns

Could this be replaced with a boolean isSidebarCollapsed ? Just trying to understand the need for 2 fields.

One of the reasons for having 2 fields is that some sidebar events in a sessions could get missed. For example, if this was in a single column we could get multiple instances of ‘uncollapsed’ and the likely reason for that would be because a ‘collapsed’ event was missed (and the analyst would have to interpret that this happened). Having it in 2 separate columns could avoid any potential data quality issues. Secondly, the analysis becomes slightly easier when the data is in a columnar format (old vs new) than having to derive it from row-wise data.
However, we can change this based on impact to engineering effort.

Next Steps:

1. @ovasileva to follow up with @Jdlrobson on effort.
2. @Mayakp.wiki to update instrumentation doc once all the above changes are confirmed.

Change 604486 abandoned by Jdlrobson:
Use packageFiles in ext.wikimediaEvents

Reason:
taking the mw.wikimediaEvents approach.

https://gerrit.wikimedia.org/r/604486

Change 604487 abandoned by Jdlrobson:
Edit count bucket pulled out of mobileWebUIActions schema

Reason:
taking the mw.wikimediaEvents approach.

https://gerrit.wikimedia.org/r/604487

Change 604492 abandoned by Jdlrobson:
Sidebar instrumentation

Reason:
taking the mw.wikimediaEvents approach.

https://gerrit.wikimedia.org/r/604492

Change 605658 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@master] Edit count bucket pulled out of mobileWebUIActions schema

https://gerrit.wikimedia.org/r/605658

Change 605659 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/WikimediaEvents@master] Sidebar instrumentation

https://gerrit.wikimedia.org/r/605659

Change 605658 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Edit count bucket pulled out of mobileWebUIActions schema

https://gerrit.wikimedia.org/r/605658

I will review this today

Change 605659 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Sidebar instrumentation

https://gerrit.wikimedia.org/r/605659

Change 607136 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Enable click tracking in Vector on beta cluster

https://gerrit.wikimedia.org/r/607136

Initial sampling rate: 20%. Sampling rate should be configurable so we can change it later if necessary.

We are adding ability to do this dynamically just this week to event streams that use the new schema repo and configuration, from the client, work is much the same as the client is also eventlogging, from the backend you would just need to create 1 file where your config is specified for sampling and you can change it dynamically without deploying code

Pinging @mpopov and @jlinehan so they can help on this regard.

plus the # of events/sec are manageable with the legacy system given that we are only going to deploy this to few small/medium test wikis .. which was the main consideration for using existing or new EL system.

Correction: both systems work the same when it comes to scaling, The advantage of MEP for this project is not scale but rather dynamic configuration for sampling. Also, instrumentation is same, both use the eventlogging client. For MEP just the schema does not reside on Meta wiki.

Correction: both systems work the same when it comes to scaling

Sorta? eventlogging-processor needs hardware provisioning and manual configuration to scale, eventgate scales with more k8s pods.

I hope to get all the EL docs updated very soon.

FYI, some intrumentation docs here: https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To

Change 607136 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable click tracking in Vector on beta cluster

https://gerrit.wikimedia.org/r/607136

I reckon the guidelines document is still private: https://docs.google.com/spreadsheets/d/1gf0gHuo8pXUZDL8aQDrf7QdWBn-dM2tCG4tGZIEBuoU/edit#gid=0
Now that it's finalized can this be published for the benefit of volunteers?

In T250282#6254750, @Demian wrote:

I reckon the guidelines document is still private: https://docs.google.com/spreadsheets/d/1gf0gHuo8pXUZDL8aQDrf7QdWBn-dM2tCG4tGZIEBuoU/edit#gid=0
Now that it's finalized can this be published for the benefit of volunteers?

@Demian - we're still finishing up a few things here - when ready, it will be available in the Schema namespace on meta - we will add a link to this task as well

@Nuria , I did reach out to @Ottomata back in May when we were designing the new schema. I would like to clarify that after discussions with @ovasileva and the web team's engineers we eventually decided to go with legacy EL as mentioned in my earlier comment-

In T250282#6145784, @Mayakp.wiki wrote:

Spoke with @ovasileva during our 1:1 on the proposal to use Modern Event Platform :
The decision is Web team would like to use Legacy EL for setting up this schema. they want to wrap up the instrumentation quickly and using a new system would have a small learning curve and engineers would need more collaboration. They're aware that eventually everything is gonna be migrated over to MEP and are fine with waiting.

plus the # of events/sec are manageable with the legacy system given that we are only going to deploy this to few small/medium test wikis .. which was the main consideration for using existing or new EL system.

Confirmed with @Jdlrobson that we have a configurable sampling rate; set to 0 on production and 100% on beta cluster right now. The schema is on meta-wiki : https://meta.wikimedia.org/wiki/Schema:DesktopWebUIActionsTracking

June 25, 2020: new Open question have come up

1. name of "donate" link

Every sidebar is different depending on the wiki. It’s defined by https://en.wikipedia.org/wiki/MediaWiki:Sidebar . As you can see there donate is defined as “sitesupport-url|sitesupport” which means we will see name=n-sitesupport for this specific link. This is not an ideal value to be stored in the database table.
Action: @Mayakp.wiki @has asked if we can transform the name. Jdlrobson is seeing if it's possible to change just the donate link and at what cost.

2. Sidebar instrumentation gotcha - any community liaison support needed?

We will be discussing on email to get to a consensus and then posting updates here, if any.

Is it possible to get general click statistics for each wiki as part of this event? It will be convenient for the community to properly configure their sidebars. At the moment, it is important for the legacy vector.

Discussed the below items in our instrumentation meeting today:

In T250282#6258514, @Mayakp.wiki wrote:

June 25, 2020: new Open question have come up

1. name of "donate" link

Every sidebar is different depending on the wiki. It’s defined by https://en.wikipedia.org/wiki/MediaWiki:Sidebar . As you can see there donate is defined as “sitesupport-url|sitesupport” which means we will see name=n-sitesupport for this specific link. This is not an ideal value to be stored in the database table.
Action: @Mayakp.wiki @has asked if we can transform the name. Jdlrobson is seeing if it's possible to change just the donate link and at what cost.

It would be best to not transform it from an engineering standpoint. Transforming the name on the client is an added cost to the user (in terms of additional bytes of JavaScript) and potential for false positives.
Question : Will the 'donate' link always show up as 'n-sitesupport' or will there be other values that will need to be interpreted as donate? Jon will look at the target wikis we plan to run analysis on to see if they all use n-sitesupport.
Alternatively, we could think about creating a data dictionary for this schema.. more on that later.

2. Sidebar instrumentation gotcha - any community liaison support needed?

We will be discussing on email to get to a consensus and then posting updates here, if any.

These types of changes are actually quite rare and it's unlikely we'll encounter one during the time we're measuring. That said, as the communities are still (sort of) the owners of the contents of the sidebar we will be aware of any changes being discussed, in advance. Product Manager will inform the Analyst of any potential changes to the sidebar links and discuss how it will impact analysis.

Question : Will the 'donate' link always show up as 'n-sitesupport' or will there be other values that will need to be interpreted as donate? Jon will look at the target wikis we plan to run analysis on to see if they all use n-sitesupport.

I've checked and on all the test wikis its n-sitesupport
Note: Office Wiki has no donate link and the position of the donate link changes - for example on Basque Wikipedia it's in the third menu down, rather than the first menu.

Test Result - Beta

Status: ✅ PASS
Environment: beta
OS: macOS Catalina
Browser: Chrome
Device: MBP
Emulated Device: iPhone 11 Pro Max

Test Artifact(s):

QA steps

✅ AC1: Visit https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page and verify an init event is fired when the page unloads (you'll need to click 'preserve network requests')

{"event":{"action":"init","isAnon":true,"skinVersion":2,"skin":"vector","editCountBucket":"0 edits","isSidebarCollapsed":false,"token":"85a0da76eac74d2851eb"},"schema":"DesktopWebUIActionsTracking","webHost":"en.wikipedia.beta.wmflabs.org","wiki":"enwiki","revision":20184974};:

✅ AC2: Click a link in the sidebar and verify a click event is fired when the page unloads (you'll need to click 'preserve network requests')

{"event":{"action":"click","isAnon":true,"skinVersion":2,"skin":"vector","editCountBucket":"0 edits","isSidebarCollapsed":false,"token":"85a0da76eac74d2851eb","name":"n-randompage"},"schema":"DesktopWebUIActionsTracking","webHost":"en.wikipedia.beta.wmflabs.org","wiki":"enwiki","revision":20184974};:

✅ AC3: Click the toggle menu button and verify a click event is fired when the page unloads (you'll need to click 'preserve network requests')

When I collapse the sidebar menu:

{"event":{"action":"click","isAnon":true,"skinVersion":2,"skin":"vector","editCountBucket":"0 edits","isSidebarCollapsed":true,"token":"85a0da76eac74d2851eb","name":"ui.sidebar"},"schema":"DesktopWebUIActionsTracking","webHost":"en.wikipedia.beta.wmflabs.org","wiki":"enwiki","revision":20184974};:

When I expand the sidebar menu:

{"event":{"action":"click","isAnon":true,"skinVersion":2,"skin":"vector","editCountBucket":"0 edits","isSidebarCollapsed":false,"token":"85a0da76eac74d2851eb","name":"ui.sidebar"},"schema":"DesktopWebUIActionsTracking","webHost":"en.wikipedia.beta.wmflabs.org","wiki":"enwiki","revision":20184974};:

✅ AC4: Verify an init event is fired in legacy Vector when the page unloads (you'll need to click 'preserve network requests')

{"event":{"action":"init","isAnon":false,"skinVersion":1,"skin":"vector","editCountBucket":"5-99 edits","isSidebarCollapsed":false,"token":"85a0da76eac74d2851eb"},"schema":"DesktopWebUIActionsTracking","webHost":"en.wikipedia.beta.wmflabs.org","wiki":"enwiki","revision":20184974};:

✅ AC5: Verify a click event is fired in legacy Vector when the page unloads (you'll need to click 'preserve network requests')

{"event":{"action":"click","isAnon":false,"skinVersion":1,"skin":"vector","editCountBucket":"5-99 edits","isSidebarCollapsed":false,"token":"85a0da76eac74d2851eb","name":"n-randompage"},"schema":"DesktopWebUIActionsTracking","webHost":"en.wikipedia.beta.wmflabs.org","wiki":"enwiki","revision":20184974};:

@Edtadros per our conversation I've created a reminder to enable sidebar instrumentation on test.wikipedia so you can test this in production. This will be blocked until next week.

Thanks!