@jwang - As a reference for this task, I created a doc to help clarify the instrumentation added to the ULS schema to track the AB events. I've added this doc to the task description as well.

The doc includes the following worksheets:

AB Test Events Sheet: Events that are expected to be sent for the control and treatment groups in the AB test as well as the filters needed to isolate events to sessions in the AB test.
Relevant Tasks: Links to all the relevant instrumentation tasks
Language Switcher Screenshot Screenshot of the language setting in the sidebar (control) and new language button in the header (control) group.

Hope this helps but please feel free to reach out if you have any questions.

Thank to Megan for the detailed doc. While learning this doc, I also created a slide deck to document the schema examples of each type of events.

Regarding the "Analysis Criteria: Which group has a higher rate of language switching?", I am wondering what the definition of the rate could be. Based my current knowledge of the data, it seems that we cannot calculate a click through rate here. If the rate is "the clicks of language link / number of language window initializations", we don't have the data of the number of language window initializations for control group. If rate is defined as "the clicks of language link / number of pageviews", given the web_pageview_id field is null in the database, it seems we can not have the number of pageviews for both control and treatment groups. Hope to discuss further in our next meeting.

Olga, Megan and I have discussed the metrics for A/B testing analysis in the meeting. It's confirmed that we cannot measure "rate of language switching" with current instrumentation. We'd like to measure the number of language switch events if we can confirm the bucketing for the languages A/B tests worked as expected. The confirmation task is tracked by ticket T286932.

In T285755#7303323, @jwang wrote:

Olga, Megan and I have discussed the metrics for A/B testing analysis in the meeting. It's confirmed that we cannot measure "rate of language switching" with current instrumentation. We'd like to measure the number of language switch events if we can confirm the bucketing for the languages A/B tests worked as expected. The confirmation task is tracked by ticket T286932.

Sounds good, thanks @jwang! Left a note on T286932: Confirm the bucketing for the languages A/B tests worked as expected. Bucketing looks good from our side, but was wondering if you can take a look for final confirmation.

AB testing report is published at http://nbviewer.org/github/jenniferwang-wmf/Web_language_switch/blob/master/Language_switching_login_user_AB_test_report.ipynb.

Summary:

• language link clicks

There was an average 44.57% decrease in total clicks on language links by logged-in users on the early adopter wikis in treatment group compared with control group. The decrease is contradicted with our hypothesis. It might be due to the lags in new feature adoption. Further investigation may be needed to identify possible reasons for this decrease. R&D is double checking the instrumentation now (https://phabricator.wikimedia.org/T291285)

• input language link clicks

There was an average 134.75% increase in total clicks on input language links by logged-in users on the early adopter wikis in treatment group (with deployment of the new language feature). The average is driven upwards mainly by wikis with low activities.
The activities of input language switch is low cross wikis, too less to statistically model the impact.

• interface language click clicks

There was an average 36.9% decrease1 in total clicks on interface language links by logged-in users on the early adopter wikis in treatment group (with deployment of the new language feature). The average is driven downwards mainly by wikis with low activities.

The activities of interface language switch is low cross wikis, too less to statistically model the impact.

• How frequently do users open the list but not switch to a language?

In average1, 30.9% of new button clicks do not lead to the next step: switch to a language. If exclude two low activity wikis, German Wikivoyage and Portuguese Wikiversity, the average non-switch rate is 35.4%.