Page MenuHomePhabricator

Analyze results of A/B test for new language switching location
Open, HighPublic

Description

Description

In T269093 we deployed an A/B test that compares the old location of language switching versus the new for logged in users on the pilot wikis for the desktop improvements project.

Analysis Criteria

Which group has a higher rate of language switching?
How frequently do users open the list but not switch to a language?
Have any other interesting trends emerged?
(for logged-out users) Is there any perceived changes in search behavior before/after the change?

Background Details

AB Instrumentation Details
Deployment Dates:
AB Start Dates:
June 22nd: frwiktionary, hewiki, ptwikiversity, frwiki, euwiki, ptwiki, kowiki, trwiki, srwiki, bnwiki, dewikivoyage, vecwiki
June 28th: fawiki

AB Test End Date:
July 20th : All test wikis

Event Timeline

@jwang - As a reference for this task, I created a doc to help clarify the instrumentation added to the ULS schema to track the AB events. I've added this doc to the task description as well.

The doc includes the following worksheets:

AB Test Events Sheet: Events that are expected to be sent for the control and treatment groups in the AB test as well as the filters needed to isolate events to sessions in the AB test.
Relevant Tasks: Links to all the relevant instrumentation tasks
Language Switcher Screenshot Screenshot of the language setting in the sidebar (control) and new language button in the header (control) group.

Hope this helps but please feel free to reach out if you have any questions.

Thank to Megan for the detailed doc. While learning this doc, I also created a slide deck to document the schema examples of each type of events.

Regarding the "Analysis Criteria: Which group has a higher rate of language switching?", I am wondering what the definition of the rate could be. Based my current knowledge of the data, it seems that we cannot calculate a click through rate here. If the rate is "the clicks of language link / number of language window initializations", we don't have the data of the number of language window initializations for control group. If rate is defined as "the clicks of language link / number of pageviews", given the web_pageview_id field is null in the database, it seems we can not have the number of pageviews for both control and treatment groups. Hope to discuss further in our next meeting.

Olga, Megan and I have discussed the metrics for A/B testing analysis in the meeting. It's confirmed that we cannot measure "rate of language switching" with current instrumentation. We'd like to measure the number of language switch events if we can confirm the bucketing for the languages A/B tests worked as expected. The confirmation task is tracked by ticket T286932.

Olga, Megan and I have discussed the metrics for A/B testing analysis in the meeting. It's confirmed that we cannot measure "rate of language switching" with current instrumentation. We'd like to measure the number of language switch events if we can confirm the bucketing for the languages A/B tests worked as expected. The confirmation task is tracked by ticket T286932.

Sounds good, thanks @jwang! Left a note on T286932: Confirm the bucketing for the languages A/B tests worked as expected. Bucketing looks good from our side, but was wondering if you can take a look for final confirmation.