Analyze Language Switching Usage trend on anonymous users 3 months after new language button deployed
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jwang
	Oct 26 2021, 8:57 PM

Description

Background

New language switch has been enabled to anonymous user on June 22, 2021 on all pilot wikis. We have analyzed the impact of new language switch on-by-default on anonymous user's language switch behavior. T289200. The analysis covered timeframe from June 11, 2021 to August 31. 2021.

While double checking the instrumentation to find the potential reason of usage drop, R&D found the instrumentation was not enabled on new feature button currently. T291285

Request

@ovasileva request to look at the recent trend of new language button clicks by wikis, to understand when instrumentation changed.

Timeframe to be analyzed

Start Date: ~ (the earliest data we have)
End Date: 2021-10-31

Participating wikis

frwiktionary, hewiki, ptwikiversity, frwiki, euwiki, ptwiki, kowiki, trwiki, srwiki, bnwiki, dewikivoyage, vecwiki, fawiki

Done

Data sanity check T294390#7464117
Analyze Language Switching Usage trend on anonymous users 3 months after deployment. T294390#7501104
- New language button clicks
- Language link clicks after new button clicks
- Input language link clicks after new button clicks

Report: https://nbviewer.org/github/jenniferwang-wmf/Web_language_switch/blob/master/Language_switching_logout_user_4month_after_deployment_report.ipynb

Decision to be made

Related Objects
Search...

Status	Assigned	Task
Resolved	ovasileva	T275766 Log sessionID, isAnon, editBucketCount, and selectedLanguage properties in the UniversalLanguageSelector instrument
Resolved	ovasileva	T275794 Log time on page before switching language as part of the UniversalLanguageSelector instrument
Resolved	ovasileva	T256023 [EPIC] Improve language switching capabilities
Resolved	jwang	T285755 Analyze results of A/B test for new language switching location
Resolved	jwang	T289200 Analyze behavior change on logged-out users before/after turn-on new language switch
Open	ovasileva	T297367 [EPIC] Iterate on language button to improve discoverability
Resolved	ovasileva	T291286 Explore iterations of current language button implementation: Make language button progressive
Resolved	jwang	T294390 Analyze Language Switching Usage trend on anonymous users 3 months after new language button deployed

Event Timeline

jwang created this task.Oct 26 2021, 8:57 PM

Restricted Application added subscribers: Huji, revi, Aklapper. · View Herald TranscriptOct 26 2021, 8:57 PM

jwang renamed this task from Analyze Language Switching Usage trend on anonymous user 3 months after deployment to Analyze Language Switching Usage trend on anonymous user 3 months after new language button deployed.Oct 26 2021, 9:53 PM

jwang renamed this task from Analyze Language Switching Usage trend on anonymous user 3 months after new language button deployed to Analyze Language Switching Usage trend on anonymous users 3 months after new language button deployed.Oct 27 2021, 5:07 PM

Data sanity check

The trend shows the new button click events dropped since 2021-09-16. The number of events recovered since 2021-10-14. It has recovered to the level before the drop.

button_date	wiki	new_button_clicks
2021-09-14	frwiki	93272
2021-09-15	frwiki	93204
2021-09-16	frwiki	75324
2021-09-17	frwiki	1277
2021-09-18	frwiki	625
2021-09-19	frwiki	81
2021-09-20	frwiki	37
2021-09-21	frwiki	23
2021-09-22	frwiki	6
2021-09-23	frwiki	11
2021-09-24	frwiki	7
2021-09-25	frwiki	4
2021-09-26	frwiki	5
2021-09-28	frwiki	2
2021-09-29	frwiki	1
2021-10-01	frwiki	1
2021-10-02	frwiki	5
2021-10-06	frwiki	1
2021-10-07	frwiki	2
2021-10-10	frwiki	1
2021-10-12	frwiki	2
2021-10-14	frwiki	14526
2021-10-15	frwiki	94008

jwang updated the task description. (Show Details)Oct 28 2021, 5:45 AM

@jwang - Thanks for looking into this! I think this was due to the bug introduced that @cjming tracked down. Would it be possible to get a graph of language switching events before/after the change up to the current week (similar to T289200#7356197)?

ovasileva added parent tasks: T289200: Analyze behavior change on logged-out users before/after turn-on new language switch, T291286: Explore iterations of current language button implementation: Make language button progressive.Oct 28 2021, 10:04 AM

jwang moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Nov 9 2021, 6:05 PM

jwang updated the task description. (Show Details)Nov 10 2021, 7:02 PM

Summary

New Language Button Clicks

The new button click events have dropped since 2021-09-16. The number of events recovered since 2021-10-14. It has recovered to the level before the drop as of 2021-10-31. Engineer has tracked down the instrumentation issue in T291285.

Language Link Clicks

Comparing the recent week (2021-10-25 ~ 2021-10-31) after the deployment and data recovery to the week before the deployment, there was an average [1] 50.15% decrease in total weekly clicks on language links by logged-out users on the early adopter wikis. In the previous report, the decrease in the week of the 16th day to the 23th day after the deployment is 61.1%.

It seems new feature adoption is catching up slowly.
[1]: Calculated by taking the average of the percent changes observed on each early adopter wiki.

Input Language Link Clicks

In the previous report, there was an average 69.9% increase in daily average clicks on input language links by logged-out users on the early adopter wikis following deployment of the new language feature. The average is driven upwards mainly by several wikis with low daily activities.

The recent trend shows:

The number of clicks dropped between 2021-09-16 and 2021-10-14 due to the instrumentation issue mentioned at the beginning.
Now the number of clicks has recovered to the level before the issue was introduced.
The trend of input language clicks is unclear due to the low activity on most pilot wikis.

Report is posted at: https://nbviewer.org/github/jenniferwang-wmf/Web_language_switch/blob/master/Language_switching_logout_user_4month_after_deployment_report.ipynb

Thank you @jwang! Based on this, we're ready to continue with T291286: Explore iterations of current language button implementation: Make language button progressive and T295555: Language switching: put an alert in the sidebar about where the language links are

ovasileva mentioned this in T291286: Explore iterations of current language button implementation: Make language button progressive.Nov 15 2021, 10:45 AM

jwang moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.Nov 16 2021, 6:05 PM

ovasileva closed this task as Resolved.Nov 16 2021, 7:15 PM

ovasileva mentioned this in T289200: Analyze behavior change on logged-out users before/after turn-on new language switch.

	F34744150: image.png
	Nov 12 2021, 9:52 PM

	F34744152: image.png
	Nov 12 2021, 9:52 PM

	F34744148: image.png
	Nov 12 2021, 9:52 PM

	F34744145: image.png
	Nov 12 2021, 9:52 PM

Analyze Language Switching Usage trend on anonymous users 3 months after new language button deployedClosed, ResolvedPublicActions