@ovasileva I completed post-deployment QA of the instrumentation added to track clicks to the languages list in the sidebar. T275762. All events appear to be recording as expected.

Summary of a few observations:-

• We start recording clicks to the sidebar language links on 28 April 2021. There are an average 176,447 sessions per day (across all wikis and all user types). No unexpected spikes or drops so far.
• Both logged-in and anonymous users typically click on a language link only once in a session. The maximum number of clicks in a session recorded for logged-in users was 2 and for logged-out users was 4.
• Almost all of the language link clicks recorded to date (99% of all sessions) have been on the test wikis. I believe this is expected as the new language switcher button was deployed to all users opt'd in to the modern vector on all non test wikis. Users with modern vector on the test wikis have not been shown the new language switcher and still shown the language links in the sidebar. Also, it looks like instrumentation was limited to the legacy sidebar on modern vector so we are not recording clicks by legacy vector users or from other desktop skin types.
• The most frequent language switches using the language links are to English (73% of sessions) followed by Spanish (6.5%), and German (6.3%).
• French was the most common initial language (56% of all sessions). This is partly due to it being one of the test wikis where we are predominately recording clicks to the links in the sidebar.

See QA notebook for further details.

One additional observation following QA of clicks to the language list in the sidebar:

About 0.1% of all sessions with clicks to the language list in the sidebar came from logged-in users on modern vector on a non-test wiki. All of these users should be shown the language switcher button and not shown the language list in the sidebar. I'm not sure what circumstances cause these events to occur; however, since it represents such a small percent of sessions it should not impact analysis of the overall trends.

@MNeisler - T281928: No event fired for clicking the language button and T280825: Create harness for A/B test for logged in users are now ready for testing

Working on QA of the new language switcher button instrumented in T281928.

A few initial observations

Needs further investigation or clarification:

• About 8.6% of all sessions with clicks to the new lang button (logged as event.action = 'compact-language-links-open'). were recorded on the test wikis (8.1% of these sessions were from logged-out users and only 0.3% from logged-in users). For non-test wikis, 97% of sessions with clicks to the new lang button were recorded by logged-out users. See table below. @ovasileva Does this seem possible based on current config states?

New Language Button Clicks from 14 May 2021 through 24 May 2021

• Only about 3.4% of sessions where the language button was clicked were followed by an event to switch the language (event.action = 'language-change'). This seems smaller than expected. I did some clicking around on English Wiki and I saw events fire when I clicked the button but not when I clicked a language link under the button. @phuedx - Possible there is a sampling issue occurring here?

Passed checks:

• Confirmed we are logging language button clicks. From the deployment of instrumentation to today (14 May through 24 May 2021), we've logged a total of 508,077 unique sessions with a click to the language button and 600,955 clicks to the language button.
• We are recording events across all wikis: both test and non-test wikis. Almost half (49.7%) of sessions with a click to the language button were recorded on English Wikipedia, followed by Russian, German and Spanish Wikipedia.
• Confirmed we are able to break down language button click events by user edit count
• Confirmed we are recoding multiple clicks (the first click and subsequent clicks) to the button within a single session. There is an average of 1 to 2 button clicks per session along with some outliers with over 500 clicks a session which are likely a sign of automated traffic.
• After a user clicks the language button, there are two event.context events recorded: interface (user changed interface language) and languages-list (user clicks lang links that appear underneath the button). Almost all recorded language switches following a click to the language button were to 'event.context = ''languages-list'`, which seems expected. (Note: This is also how we record clicks to the links in the sidebar however I can look for sessions that include clicks to the language button to distinguish users in the two groups.

See notebook for details.

In T280770#7110028, @MNeisler wrote:

Working on QA of the new language switcher button instrumented in T281928.

A few initial observations

Needs further investigation or clarification:

• About 8.6% of all sessions with clicks to the new lang button (logged as event.action = 'compact-language-links-open'). were recorded on the test wikis (8.1% of these sessions were from logged-out users and only 0.3% from logged-in users). For non-test wikis, 97% of sessions with clicks to the new lang button were recorded by logged-out users. See table below. @ovasileva Does this seem possible based on current config states?

Confirming that this is not expected. Currently the language button should not be showing to any logged-out users. There is a possibility of viewing the button using the parameter from T282543: Add parameter that allows language button to be seen across all wikis but that would be probably <100 clicks at most.

In T280770#7110028, @MNeisler wrote:

Needs further investigation or clarification:

• About 8.6% of all sessions with clicks to the new lang button (logged as event.action = 'compact-language-links-open'). were recorded on the test wikis (8.1% of these sessions were from logged-out users and only 0.3% from logged-in users). For non-test wikis, 97% of sessions with clicks to the new lang button were recorded by logged-out users. See table below. @ovasileva Does this seem possible based on current config states?

It is possible. Logged out users using Vector V1 can still open the version of the compact language links dialog that is triggered by the "N more" button at the bottom of the language list in the sidebar. We can make trivial changes to the ULS and the instrument to provide more context for these events so that you can filter out those that you're not interested in during analysis.

In T280770#7115368, @phuedx wrote:

In T280770#7110028, @MNeisler wrote:

Needs further investigation or clarification:

• About 8.6% of all sessions with clicks to the new lang button (logged as event.action = 'compact-language-links-open'). were recorded on the test wikis (8.1% of these sessions were from logged-out users and only 0.3% from logged-in users). For non-test wikis, 97% of sessions with clicks to the new lang button were recorded by logged-out users. See table below. @ovasileva Does this seem possible based on current config states?

It is possible. Logged out users using Vector V1 can still open the version of the compact language links dialog that is triggered by the "N more" button at the bottom of the language list in the sidebar. We can make trivial changes to the ULS and the instrument to provide more context for these events so that you can filter out those that you're not interested in during analysis.

Does this mean that in analysis we are able to currently distinguish between these three?

• People selecting language button on top of the page
• People that are in the control bucket clicking the "more" button
• People that have opted out of modern vector clicking the more button

Change 695351 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/UniversalLanguageSelector@master] Pass context to compact_language_links.open hook

https://gerrit.wikimedia.org/r/695351

Change 695353 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/WikimediaEvents@master] universalLanguageSelector: Add missing properties

https://gerrit.wikimedia.org/r/695353

I discussed this with @ovasileva and we concluded that we can backport that above changes next week and then wait until the week after to enable the A/B test.

Change 695351 merged by jenkins-bot:

[mediawiki/extensions/UniversalLanguageSelector@master] Pass context to compact_language_links.open hook

https://gerrit.wikimedia.org/r/695351

Change 695353 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] universalLanguageSelector: Add missing properties

https://gerrit.wikimedia.org/r/695353

Change 698535 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/WikimediaEvents@wmf/1.37.0-wmf.7] universalLanguageSelector: Add missing properties

https://gerrit.wikimedia.org/r/698535

Change 698536 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.8] Pass context to compact_language_links.open hook

https://gerrit.wikimedia.org/r/698536

Change 698536 abandoned by Phuedx:

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.8] Pass context to compact_language_links.open hook

Reason:

Wrong branch...

https://gerrit.wikimedia.org/r/698536

Change 698537 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.7] Pass context to compact_language_links.open hook

https://gerrit.wikimedia.org/r/698537

Change 698537 merged by jenkins-bot:

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.7] Pass context to compact_language_links.open hook

https://gerrit.wikimedia.org/r/698537

Change 698535 merged by Urbanecm:

[mediawiki/extensions/WikimediaEvents@wmf/1.37.0-wmf.7] universalLanguageSelector: Add missing properties

https://gerrit.wikimedia.org/r/698535

Mentioned in SAL (#wikimedia-operations) [2021-06-08T11:31:59Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/UniversalLanguageSelector/resources/js/ext.uls.launch.js: 5df13eeae3b52b98eaf3fdb99ddfa5a0f7b2b1e4: Pass context to compact_language_links.open hook (T280770) (duration: 00m 57s)

Mentioned in SAL (#wikimedia-operations) [2021-06-08T11:37:19Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/WikimediaEvents/: b0b46530b731d2a5f17b0aa04a4cf99df175e23d: universalLanguageSelector: Add missing properties (T280770) (duration: 00m 56s)

@ovasileva @phuedx - Finished post-deployment QA of the new ULS changes and identified one possible bug or missing instrumentation. Everything else looks good. See details below.

POSSIBLE BUG/MISSING INSTRUMENTATION:

• For sessions with clicks to the new language button in the header, I'm not seeing any language change events (event.action = 'language-change') for when a user changes language by clicking on one of the language links that appear after the user clicks the button. Was instrumentation added to track these events?
  
  If not, it will be difficult to accurately compare language switches between the two test groups. We will be able to compare clicks to open the new language button in the treatment group to clicks to the sidebar by the control group (including lang links in sidebar, N more button, and ULS settings). It looks like the 'display' and 'input' settings for the new button were instrumented so we will also be able to compare changes to these settings but we're losing details on if a user that clicks the new button ends up actually switches languages.

PASSED CHECKS:

• Confirmed we are now logging event.skin, event.skinVersion, and additional event.context fields in the ULS schema starting on 8 June 2021.
• Confirmed we are currently only recording clicks to the new language button by logged-in users on non-test wikis as expected.
• Confirmed that with new instrumentation we can differentiate the following event types:
  • People on vector selecting language button on top of the page: event.action = 'compact-language-links-open'; event.context = 'header'; event.skinVersion = 'latest'
  • People on vector that are in the control bucket clicking the " N more" button in the sidebar: event.action = 'compact-language-links-open' and event.context = 'other', event.skinVersion = 'latest'
  • People that have opted out of modern vector clicking the more button: event.context = 'NULL'; event.skinVersion = 'NULL'. There are currently no values to specify the event.context , event.skin, or event.skinVersion for non-vector skins but this still allows us to identify and filter out any non-vector clicks for the purpose of the AB test.
• New language button clicks have only been recorded on latest vector.
• Associated edit count bucket, interface and content language events are logging as expected.
• Confirmed we are logging changes to various ULS settings after the user clicks the new language button such as 'ime-change' (user changed the input method), 'no-search-results' (User searched for a language with no results) and 'webfonts-enable' (webfonts-enable). User enabled the webfonts functionality via ULS settings).

See QA notebook for further details on queries and checks

Change 699153 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/UniversalLanguageSelector@master] Fire language change hook

https://gerrit.wikimedia.org/r/699153

Change 699153 merged by jenkins-bot:

[mediawiki/extensions/UniversalLanguageSelector@master] Fire language change hook

https://gerrit.wikimedia.org/r/699153

Change 699183 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.9] Fire language change hook

https://gerrit.wikimedia.org/r/699183

Change 699183 merged by jenkins-bot:

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.9] Fire language change hook

https://gerrit.wikimedia.org/r/699183

Mentioned in SAL (#wikimedia-operations) [2021-06-10T18:38:40Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.9/extensions/UniversalLanguageSelector/resources/js/ext.uls.launch.js: 8aeab139879613782548b20fc11af5e66589e30a: Fire language change hook (T280770) (duration: 01m 07s)

In T280770#7147624, @MNeisler wrote:

POSSIBLE BUG/MISSING INSTRUMENTATION:

• For sessions with clicks to the new language button in the header, I'm not seeing any language change events (event.action = 'language-change') for when a user changes language by clicking on one of the language links that appear after the user clicks the button. Was instrumentation added to track these events?

Regrettably, no. Neither version of the language switcher treatment was instrumented.

The above patch fixes this and was deployed everywhere yesterday evening (BST). However, there is a caveat: it's quite difficult to thread through a value for event.context so it'll always be the default ("interface"), i.e. if the user has opened the language switcher in the header and switched language, we should two events with the following properties:

1. event.web_session_id="foo" event.action="compact-language-links-open" event.context="header" event.skinversion="latest"
2. event.web_session_id="foo" event.action="language-change" event.context="interface"`

Thanks @phuedx! I confirmed that we are now logging clicks to switch languages by sessions with clicks to the new language button in the header. Data looks good to continue with the deployment of the AB test pending resolution of the open question below. (cc @ovasileva)

OPEN QUESTION:

However, there is a caveat: it's quite difficult to thread through a value for event.context through a value so it'll always be the default ("interface"),

@phuedx To confirm, this means that both of the following events are recorded the same way, correct?:

• A user clicks the language button in the header and changes the display language setting
• A user clicks the language button in the header and switches languages

This won't allow us to differentiate and compare these two event types in the AB analysis; however, we can compare the overall language switching frequency between the two groups including both clicks to change language and clicks to change display languages. Just want to confirm if this is ok for the purpose of the AB test?

Additional minor bugs/observations below (These should not impact AB test):

(1) I identified three latest vector sessions recorded as a logged-out user clicking the new language button in the header, which should not be possible.

Since there have only been three instances and two were recorded on testwiki, I'm not too worried about it impacting the AB test but documenting here just in case it's worth investigating. See details below:

Data via:

SELECT 
    TO_DATE(dt) AS `date`,
    wiki,
    event.web_session_id,
    event.skin,
    event.skinVersion,
    event.isanon,
    event.context,
    event.contentlanguage,
    Count(*) AS n_events
FROM event.universallanguageselector
WHERE 
  YEAR = 2021
  AND MONTH = 06
  AND Day >= 11
  AND event.context = 'header'
  AND event.action = 'compact-language-links-open'
  AND event.isANON = TRUE
GROUP BY
  TO_DATE(dt),
  wiki,
  event.web_session_id,
  event.skin,
  event.skinVersion,
  event.isanon,
  event.context,
  event.contentlanguage

(2) We are currently recording two desktop skin types under the event.skin field: NULL and vector, where NULL represents all non-vector skin types. This is not an issue for the AB test since this still allows us to filter out all non-vector events (and we can identify latest and legacy versions with the event.skinversion field) but it might be worth eventually revising to clarify the non-vector skin types if feasible for future analyses using this schema.

QA Notebook

In T280770#7155428, @MNeisler wrote:

However, there is a caveat: it's quite difficult to thread through a value for event.context through a value so it'll always be the default ("interface"),

@phuedx To confirm, this means that both of the following events are recorded the same way, correct?:

• A user clicks the language button in the header and changes the display language setting
• A user clicks the language button in the header and switches languages

At the moment, yes.

This won't allow us to differentiate and compare these two event types in the AB analysis; however, we can compare the overall language switching frequency between the two groups including both clicks to change language and clicks to change display languages. Just want to confirm if this is ok for the purpose of the AB test?

@ovasileva? I've just realised there is a way to thread a value for event.context through so I'm happy to make a change, if necessary.

In T280770#7157435, @phuedx wrote:

In T280770#7155428, @MNeisler wrote:

However, there is a caveat: it's quite difficult to thread through a value for event.context through a value so it'll always be the default ("interface"),

@phuedx To confirm, this means that both of the following events are recorded the same way, correct?:

• A user clicks the language button in the header and changes the display language setting
• A user clicks the language button in the header and switches languages

At the moment, yes.

This won't allow us to differentiate and compare these two event types in the AB analysis; however, we can compare the overall language switching frequency between the two groups including both clicks to change language and clicks to change display languages. Just want to confirm if this is ok for the purpose of the AB test?

@ovasileva? I've just realised there is a way to thread a value for event.context through so I'm happy to make a change, if necessary.

@phuedx - good news! - is it a difficult change? Based on https://phabricator.wikimedia.org/T273986#6876272 it seems the language setting usage is not quite negligible. Quick question on the current implementation - can we tell the difference between selecting a language from the list and changing the settings for the control group?

In T280770#7157479, @ovasileva wrote:

Quick question on the current implementation - can we tell the difference between selecting a language from the list and changing the settings for the control group?

Yes. This is a good question as it highlights the inconsistency in instrumentation between the two treatments. I'll submit a patch ASAP.

Change 699947 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/UniversalLanguageSelector@master] launchULS: Add context to interface.language.change hook

https://gerrit.wikimedia.org/r/699947

Change 699947 merged by jenkins-bot:

[mediawiki/extensions/UniversalLanguageSelector@master] launchULS: Add context to interface.language.change hook

https://gerrit.wikimedia.org/r/699947

Change 700633 had a related patch set uploaded (by Jdrewniak; author: Phuedx):

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.10] launchULS: Add context to interface.language.change hook

https://gerrit.wikimedia.org/r/700633

Change 700633 merged by jenkins-bot:

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.10] launchULS: Add context to interface.language.change hook

https://gerrit.wikimedia.org/r/700633

Change 700730 had a related patch set uploaded (by Jdrewniak; author: Phuedx):

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.9] launchULS: Add context to interface.language.change hook

https://gerrit.wikimedia.org/r/700730

Change 700730 merged by jenkins-bot:

[mediawiki/extensions/UniversalLanguageSelector@wmf/1.37.0-wmf.9] launchULS: Add context to interface.language.change hook

https://gerrit.wikimedia.org/r/700730

Mentioned in SAL (#wikimedia-operations) [2021-06-22T11:54:56Z] <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.37.0-wmf.9/extensions/UniversalLanguageSelector/: Backport: [[gerrit:700730|launchULS: Add context to interface.language.change hook (T280770)]] (duration: 00m 57s)

I did a quick check of events logged with the new context field event.context = 'content-language-switcher' and confirmed that this event is logged correctly after a user clicks a new header button and then selects a language link to switch languages.

@phuedx - Are there any other scenarios where this event context would be logged? I'm seeing event.context = 'content-language-switcher' events logged on the legacy vector (see data below). Is this also logged if a user switches languages after selecting the N More button in the sidebar?

content language switcher events logged since 22 June 2021

Data via:

SELECT
skinversion,
test_wiki_status,
SUM(1) AS num_sessions
FROM
(
SELECT
    DISTINCT
    event.web_session_id as session_id,
    event.skinVersion as skinversion,
    IF(wiki IN ('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', 
    'euwiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki' ), 'test', 'non_test') as test_wiki_status
FROM event.universallanguageselector
WHERE
    year = 2021
    AND month = 06
    AND Day >= 22
    AND useragent.is_bot = false
    AND event.context = 'content-language-switcher'
    AND event.action = 'language-change'
) dataset
GROUP BY
    skinVersion,
    test_wiki_status

In T280770#7186635, @MNeisler wrote:

@phuedx - Are there any other scenarios where this event context would be logged? I'm seeing event.context = 'content-language-switcher' events logged on the legacy vector (see data below). Is this also logged if a user switches languages after selecting the N More button in the sidebar?

Yes. See T280770#7115368. I thought we'd discussed discarding events with event.skinVersion="legacy" but now I'm concerned that we didn't.

@phuedx Got it. Thanks for clarifying. Based on the data, it looks like we're recording events with event.skinVersion = 'legacy' for all recorded language feature interactions except for clicks to the language links in the sidebar, which is currently restricted to only the latest vector. See T275762#7032377.

For the AB analysis, this should be fine as I can filter out all legacy events using the event.skinVersion field.

I'm thinking it would be useful to add a ReadMe file to the schema repo to help document and clarify the meanings of the various context and action values. I created a task (See T285888) to update.

Quick wrap-up summary for this task - I completed a review of the aggregated data recorded so far for the AB test. I've confirmed that we are recording all expected events for both test groups and overall logged-in and logged-out user data appears as expected based on the AB deployments dates.

I am unable to accurately confirm if the buckets are balanced based on the aggregated data since the ways to initiate a session in each group are different. See T280825#7191012 for details.

Per discussions in the web meeting today, further checks to make sure the AB buckets are balanced will be completed in T280825

QA notebook.