Page MenuHomePhabricator

Analyze Language Switching Usage trend on anonymous users 3 months after new language button deployed
Closed, ResolvedPublic

Description

Background

New language switch has been enabled to anonymous user on June 22, 2021 on all pilot wikis. We have analyzed the impact of new language switch on-by-default on anonymous user's language switch behavior. T289200. The analysis covered timeframe from June 11, 2021 to August 31. 2021.

While double checking the instrumentation to find the potential reason of usage drop, R&D found the instrumentation was not enabled on new feature button currently. T291285

Request

@ovasileva request to look at the recent trend of new language button clicks by wikis, to understand when instrumentation changed.

Timeframe to be analyzed

Start Date: ~ (the earliest data we have)
End Date: 2021-10-31

Participating wikis

frwiktionary, hewiki, ptwikiversity, frwiki, euwiki, ptwiki, kowiki, trwiki, srwiki, bnwiki, dewikivoyage, vecwiki, fawiki

Done
  • Data sanity check T294390#7464117
  • Analyze Language Switching Usage trend on anonymous users 3 months after deployment. T294390#7501104
    • New language button clicks
    • Language link clicks after new button clicks
    • Input language link clicks after new button clicks

Report: https://nbviewer.org/github/jenniferwang-wmf/Web_language_switch/blob/master/Language_switching_logout_user_4month_after_deployment_report.ipynb

Decision to be made

Event Timeline

jwang renamed this task from Analyze Language Switching Usage trend on anonymous user 3 months after deployment to Analyze Language Switching Usage trend on anonymous user 3 months after new language button deployed.Oct 26 2021, 9:53 PM
jwang renamed this task from Analyze Language Switching Usage trend on anonymous user 3 months after new language button deployed to Analyze Language Switching Usage trend on anonymous users 3 months after new language button deployed.Oct 27 2021, 5:07 PM
Data sanity check

The trend shows the new button click events dropped since 2021-09-16. The number of events recovered since 2021-10-14. It has recovered to the level before the drop.

button_datewikinew_button_clicks
2021-09-14frwiki93272
2021-09-15frwiki93204
2021-09-16frwiki75324
2021-09-17frwiki1277
2021-09-18frwiki625
2021-09-19frwiki81
2021-09-20frwiki37
2021-09-21frwiki23
2021-09-22frwiki6
2021-09-23frwiki11
2021-09-24frwiki7
2021-09-25frwiki4
2021-09-26frwiki5
2021-09-28frwiki2
2021-09-29frwiki1
2021-10-01frwiki1
2021-10-02frwiki5
2021-10-06frwiki1
2021-10-07frwiki2
2021-10-10frwiki1
2021-10-12frwiki2
2021-10-14frwiki14526
2021-10-15frwiki94008
ovasileva added a subscriber: cjming.

@jwang - Thanks for looking into this! I think this was due to the bug introduced that @cjming tracked down. Would it be possible to get a graph of language switching events before/after the change up to the current week (similar to T289200#7356197)?

Summary

  • New Language Button Clicks

The new button click events have dropped since 2021-09-16. The number of events recovered since 2021-10-14. It has recovered to the level before the drop as of 2021-10-31. Engineer has tracked down the instrumentation issue in T291285.

image.png (1×1 px, 567 KB)

  • Language Link Clicks

Comparing the recent week (2021-10-25 ~ 2021-10-31) after the deployment and data recovery to the week before the deployment, there was an average [1] 50.15% decrease in total weekly clicks on language links by logged-out users on the early adopter wikis. In the previous report, the decrease in the week of the 16th day to the 23th day after the deployment is 61.1%.

It seems new feature adoption is catching up slowly.
[1]: Calculated by taking the average of the percent changes observed on each early adopter wiki.

image.png (1×1 px, 539 KB)

image.png (1×1 px, 239 KB)

  • Input Language Link Clicks

In the previous report, there was an average 69.9% increase in daily average clicks on input language links by logged-out users on the early adopter wikis following deployment of the new language feature. The average is driven upwards mainly by several wikis with low daily activities.

The recent trend shows:

  1. The number of clicks dropped between 2021-09-16 and 2021-10-14 due to the instrumentation issue mentioned at the beginning.
  2. Now the number of clicks has recovered to the level before the issue was introduced.
  3. The trend of input language clicks is unclear due to the low activity on most pilot wikis.

image.png (1×1 px, 539 KB)