Test Result - Beta

Status: ❌ Fail
Environment: beta
OS: macOS Big Sur
Browser: Chrome
Device: MBP
Emulated Device: NA

Test Artifact(s):

QA Steps

Add the following items to the sticky header as per the prototype below:

✅ AC1: Language switching button

❌ AC2: Selecting button should open the ULS
Clicking on the button doesn't appear do do anything.

⬜ AC3: Note: this will only be available for wikis that have multiple language versions. Wikis with multiple languages on a single wiki (wikdata, mediawiki, etc) will not display a language selector within their sticky header
not tested
⬜ AC4: If ULS extension is not installed, do not show the ULS button in the sticky header. Do not spend time trying to wire up the fallback dropdown menu.
not tested
⬜ AC5: Make sure the new button is click tracked via DesktopWebClickTracking by having a data-event-name attribute
not tested

I can reproduce the blank button locally, but I'm struggling to reproduce the issue with the ULS not opening. On the beta cluster both ULS buttons are missing the 'mw-interlanguage-selector' class which is necessary for the ULS to open. As far as I can tell, Vector should be providing that in the template data, so something else must be causing this issue. It could be some code in the ULS extension, or perhaps some configuration issue, but so far I haven't found anything obvious.

Change 722672 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[mediawiki/skins/Vector@master] Fix sticky header language button

https://gerrit.wikimedia.org/r/722672

❌ AC2: Selecting button should open the ULS

@Edtadros I can't replicate this
Can you please check this box on preferences is checked and try it again?

Change 722672 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Fix sticky header language button

https://gerrit.wikimedia.org/r/722672

right! The sticky header is logged in only, so no need to QA for anonymous users for this feature. The query string parameter overrides this behaviour but that won't be a problem in production.

In T289815#7395479, @Jdlrobson wrote:

Article title is incorrect, the language button appears on two lines, the language selector appears far below the sticky header.

That issue relates to T289814 so I reopened the ticket.

❓ AC5: Make sure the new button is click tracked via DesktopWebClickTracking by having a data-event-name attribute

@ovasileva @jwang clicks to the language button in the sticky header will have context "other" and clicks to the one at the top of the page will have context "header". However on wikis where the language button is in the sidebar it will still be "other". Is this sufficient to distinguish the two? if so this is a pass.

Let me make sure I have this rights:
Legacy vector on:

• Clicks to language links in the sidebar: other

Legacy vector off:

• Clicks to language links in the top of the page: header
• Clicks to language links within sticky header: other

@jwang - we will want to track clicks to language links within the sticky header and compare to levels before the change. Will this be okay?

Hi, do we have a field to capture vector on/off in universallanguageselector schema? if it is captured in the event log of language button click, in theory, it should work. But I want to explore to confirm.

If vector on/off is not captured in universallanguageselector , or captured in another schema, we need further discussion. In this case, I propose "sticky" value for below scenario, if it's feasible on engineer side.
Legacy vector off:

• Clicks to language links within sticky header: sticky

Having unique value for each type of language clicks give us a way to sanity check instrumentation when there is some bug in instrumentation.

Please let me know where are we now, I can further explore the data from there.

Hi, do we have a field to capture vector on/off in universallanguageselector schema? if it is captured in the event log of language button click, in theory, it should work. But I want to explore to confirm.

I'm not sure what you mean by Vector on/off. The universallanguageselector schema has a field skin which captures whether you are in vector or another skin say minerva/monobook/timeless. It also has a skin version field which allows you to distinguish Vector 1 and Vector 2. For Vector 1, context other means sidebar. For Vector 2, context other means sticky header.

The universallanguageselector schema should have a field web_session_id. It will be possible to join this with the user session id in the A/B test schema we are creating in T292587

Sorry for the confusion. What I meant is Legacy vector on/off from @ovasileva 's comment

Legacy vector on:

Clicks to language links in the sidebar: other
Legacy vector off:

Clicks to language links in the top of the page: header
Clicks to language links within sticky header: other

If I understand your comment correctly, we do have fields to capture legacy vector on/off in universallanguageselector schema when the language link is clicked. They are event.skin and event.skin_version. And I summarized the possible value for three scenarios below. Please confirm whether it is correct.

Scenario1: Legacy vector on, clicks to language links in the sidebar

Session will include these two events:

1. event.web_session_id="foo" event.action="compact-language-links-open" event.context="other" event.skin ="vector" event.skinversion="legacy"
2. event.web_session_id="foo" event.action="language-change" event.context="content-language-switcher" event.skin ="vector" event.skinversion="legacy"

Scenario 2: Legacy vector off, clicks to language links in the top of the page

Two events sent:

1. event.web_session_id="foo" event.action="compact-language-links-open" event.context="header" event.skin ="vector" event.skinversion="latest"
2. event.web_session_id="foo" event.action="language-change" event.context="content-language-switcher" event.skin ="vector" event.skinversion="latest"

Scenario 3: Legacy vector off, clicks to language links within sticky header: other

Two events sent:

1. event.web_session_id="foo" event.action="compact-language-links-open" event.context="other" event.skin ="vector" event.skinversion="latest"
2. event.web_session_id="foo" event.action="language-change" event.context="content-language-switcher" event.skin ="vector" event.skinversion="latest"

This plan is supposed to be working. But in language switch AB testing, we have assigned event.context="other" event.skin ="vector" event.skinversion="latest" to clicks to language links in the sidebar, mentioned in instrumentation spec D13. Same field value representing different events is not recommended.

This plan is supposed to be working. But in language switch AB testing, we have assigned event.context="other" event.skin ="vector" event.skinversion="latest" to clicks to language links in the sidebar, mentioned in instrumentation spec D13. Same field value representing different events is not recommended.

If you mean different events across different experiences, then yes, these will be different.
For this experiment for all events:

• event.context="other" event.skin ="vector" event.skinversion="legacy" will mean inside the sidebar.
• event.context="other" event.skin ="vector" event.skinversion="latest" will mean inside the sticky header
• event.context="header" event.skin ="vector" event.skinversion="latest" will mean from the header

If that's a problem we can introduce "sticky", but we'll need new task if we need to do this. I'd estimate such a task as likely being a medium (1 week delay to launch).

If we only use universallanguageselector schema for language switch on sticky header AB testing analysis, it seems working for me. You can mark it pass. But I hope everyone is aware that we cannot use this schema to measure the long-term trend of new language click on sticky header because same field value represent different events in history.

In T289815#7442824, @jwang wrote:

If we only use universallanguageselector schema for language switch on sticky header AB testing analysis, it seems working for me. You can mark it pass. But I hope everyone is aware that we cannot use this schema to measure the long-term trend of new language click on sticky header because same field value represent different events in history.

@jwang - We would be interested in long-term tracking for this. Would we be able to differentiate by the date that the changes are made?

We would be interested in long-term tracking for this. Would we be able to differentiate by the date that the changes are made?

Will we change the event logging again in the future? If not, we can exclude the time range of language switch AB test in notebook based analysis. For the self explored dashboard, we may make a header note to document the inconsistency. For the new user of universallanguageselector schema, we document the timeframe of the change on the schema page. It still possibly causes confusion sometimes, especially for the new user of the schema. But if it's the only inconsistency, we would be able to differentiate the events by the date.

In T289815#7446811, @jwang wrote:

We would be interested in long-term tracking for this. Would we be able to differentiate by the date that the changes are made?

Will we change the event logging again in the future? If not, we can exclude the time range of language switch AB test in notebook based analysis. For the self explored dashboard, we may make a header note to document the inconsistency. For the new user of universallanguageselector schema, we document the timeframe of the change on the schema page. It still possibly causes confusion sometimes, especially for the new user of the schema. But if it's the only inconsistency, we would be able to differentiate the events by the date.

Thanks @jwang. I think documenting would make sense to me. The data before the change will be valuable mainly in the beginning period of implementing the change so we can see what effect adding the language button in the header has on the overall rate of language switching. In particular, we would like to know whether how the rate of usage of the language switcher at the top of the page plus the language switcher in the sticky header compares to the rate of language switching before either change. Seems like we'll have enough to determine that.

@ovasileva, please create a ticket and assign it to me.

Test Result - Beta

Status: ❓ Need more info
Environment: beta
OS: macOS Big Sur
Browser: Chrome
Device: MBP
Emulated Device: NA

Test Artifact(s):

QA Steps

Add the following items to the sticky header as per the prototype below:

✅ AC1: Language switching button

✅ AC2: Selecting button should open the ULS

✅ AC3: Note: this will only be available for wikis that have multiple language versions. Wikis with multiple languages on a single wiki (wikdata, mediawiki, etc) will not display a language selector within their sticky header
See AC1.

✅ AC4: If ULS extension is not installed, do not show the ULS button in the sticky header. Do not spend time trying to wire up the fallback dropdown menu.

❓ AC5: Make sure the new button is click tracked via DesktopWebClickTracking by having a data-event-name attribute

@ovasileva This is the only event that comes up. I don't see a DesktopWebClickTracking event with a data-event-name attribute.

{
  "event": {
    "version": 3,
    "token": "",
    "contentLanguage": "en",
    "interfaceLanguage": "en",
    "web_session_id": "ff6a93ea90cc20959f20",
    "isAnon": false,
    "skin": "vector",
    "skinVersion": "latest",
    "action": "compact-language-links-open",
    "context": "other",
    "userEditBucket": "5-99 edits"
  },
  "schema": "UniversalLanguageSelector",
  "webHost": "en.wikipedia.beta.wmflabs.org",
  "wiki": "enwiki",
  "$schema": "/analytics/legacy/universallanguageselector/1.4.0",
  "client_dt": "2021-11-16T02:26:15.817Z",
  "meta": {
    "stream": "eventlogging_UniversalLanguageSelector",
    "domain": "en.wikipedia.beta.wmflabs.org"
  }
}

Back to needs more work to double check that language instrumentation is available within either ULS or desktop click tracking schema for the language button within the sticky header

I confirmed with @jwang that the sticky header ULS button should be logging to both DesktopWebUIActionsTracking and UniversalLanguageSelector. After testing I confirmed what @Edtadros found, that the sticky ULS wasn't logging correctly to DesktopWebUIActionsTracking due to a missing data attribute. I can put up a patch to fix that.

I'm seeing events being correctly logged to UniversalLanguageSelector as well, but the context was set to "other". I believe right now it's either going to be set to "header" or "other". @jwang Should I also update the ULS instrumentation to set a more helpful context like "sticky-header`?

@jwang Should I also update the ULS instrumentation to set a more helpful context like "sticky-header`?

Yes, context = "sticky-header` for click event on sticky header in ULS instrumentation will be very helpful for analysis.

Change 740199 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[mediawiki/extensions/WikimediaEvents@master] Update UniversalLanguageSwitcher instrumentation to handle sticky header context

https://gerrit.wikimedia.org/r/740199

Change 740200 had a related patch set uploaded (by Bernard Wang; author: Bernard Wang):

[mediawiki/skins/Vector@master] Ensure sticky header ULS is tracked by DesktopWebUIActionsTracking

https://gerrit.wikimedia.org/r/740200

Change 740200 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Ensure sticky header ULS is tracked by DesktopWebUIActionsTracking

https://gerrit.wikimedia.org/r/740200

Change 740199 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Update UniversalLanguageSwitcher instrumentation to handle sticky header context

https://gerrit.wikimedia.org/r/740199

@Jdlrobson - is AC4 a fail also? I've created T297362: Sticky header: ensure correct language presentation on multi-language wikis to track AC3

@Edtadros - this is going back to QA - we will track AC3 in a new task

@ovasileva I don't think we got the answer on AC4.

Test Result - Prod

Status: ✅
Environment: enwiki
OS: macOS Big Sur
Browser: Chrome
Device: MBP
Emulated Device: NA

Test Artifact(s):

QA Steps

Add the following items to the sticky header as per the prototype below:

✅ AC1: Language switching button

✅ AC2: Selecting button should open the ULS
See AC1

⬜ AC3: Note: this will only be available for wikis that have multiple language versions. Wikis with multiple languages on a single wiki (wikdata, mediawiki, etc) will not display a language selector within their sticky header
This will be tracked in a different task per T289815#7560806

❓ AC4: If ULS extension is not installed, do not show the ULS button in the sticky header. Do not spend time trying to wire up the fallback dropdown menu.
I'm not sure how to test the ULS not being installed, however checking a random page with only 1 language, as expected didn't show the language button in the sticky header. Let me know if this is sufficient for validating this AC.

✅ AC5: Make sure the new button is click tracked via DesktopWebClickTracking by having a data-event-name attribute

oops... I retested this. Same result....just not sure about AC4, but looks like you already answered that.