@ovasileva, here are the usage trends of new feature and old features on logged-out users. The usage data is not in line with the deployment agenda.

Usage of new feature

We observed the new feature click events since June 22, 2021 and no obvious changes around the deployment date (2021-07-20). The new feature usage adoption is not in line with the deployment agenda.

SELECT
    TO_DATE(dt) as button_date,
    wiki as wiki,
    count(1) AS new_button_clicks
FROM event.universallanguageselector
WHERE
    year = 2021 and month in (6,7,8) 
    AND (CONCAT(year,LPAD(month,2,'0'),LPAD(day,2,'0')) between '20210601' and '20210822')
    AND wiki in ('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', 
    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki')
    AND useragent.is_bot = false
-- only anonymous users
    AND event.isanon = true
    AND event.action = 'compact-language-links-open'
    AND event.context = 'header'
    AND event.skinVersion = 'latest'
GROUP BY 
    TO_DATE(dt),
    wiki

Usage of old feature -- language link on the sidebar

We observed the old feature language link on the sidebar received less clicks since June 22, 2021 and no obvious change around 2021-07-20.

code

query_control_language_link <- 
"
SELECT
    TO_DATE(dt) AS `date`,
    wiki,
    event.web_session_id,
    event.usereditbucket,
    event.timetochangelanguage,
    event.interfacelanguage,
    event.contentlanguage,
    event.selectedinterfacelanguage,
    Count(*) AS n_events
FROM event.universallanguageselector
WHERE
    year = 2021 and month in (6,7,8) 
    AND (CONCAT(year,LPAD(month,2,'0'),LPAD(day,2,'0')) between '20210601' and '20210822')
    AND event.context = 'languages-list' and event.action = 'language-change' 
    AND event.skinVersion = 'latest' 
    AND wiki in ('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', 
    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki')
    AND useragent.is_bot = false 
-- only anonymous users
    AND event.isanon =true
GROUP BY
    TO_DATE(dt),
    wiki,
    event.web_session_id,
    event.usereditbucket,
    event.timetochangelanguage,
    event.interfacelanguage,
    event.contentlanguage,
    event.selectedinterfacelanguage
"
lang_link_events <-  wmfdata::query_hive(query_control_language_link)
lang_link_events_by_date <- lang_link_events %>%
    group_by( wiki, date) %>%
    summarize(n_events_language_link = sum(n_events))

Usage of old feature -- N-more button on the sidebar

We observed the old feature n-more button on the sidebar received less clicks since June 22, 2021 and no obvious change around 2021-07-20.

code

--  N-more lang button was selected

SELECT
    TO_DATE(dt) as button_date,
    wiki as wiki, 
    count(1) AS n_more_clicks
FROM event.universallanguageselector
WHERE
    year = 2021 and month in (6,7,8) 
    AND (CONCAT(year,LPAD(month,2,'0'),LPAD(day,2,'0')) between '20210601' and '20210822')
    AND wiki in ('frwiktionary', 'hewiki', 'ptwikiversity', 'frwiki', 
    'euwiki', 'fawiki', 'ptwiki', 'kowiki', 'trwiki', 'srwiki', 'bnwiki', 'dewikivoyage', 'vecwiki')
    AND useragent.is_bot = false
-- only anonymous users
    AND event.isanon =true
    AND event.action = 'compact-language-links-open'
    AND event.context = 'other'
    AND event.skinVersion = 'latest'
GROUP BY 
    TO_DATE(dt),
    wiki

The graphs look to be time shifted by a month.

I know that in JavaScript month 0 is January. Is it possible when doing LPAD in the query we are creating the wrong date?

The month in schema starts with 1 for January. Also the data is grouped by field dt, which is the timestamp of the event.

select dt, year, month, day
FROM event.universallanguageselector
WHERE
    year = 2021 and month in (6,7,8) 
    AND (CONCAT(year,LPAD(month,2,'0'),LPAD(day,2,'0')) between '20210601' and '20210822')
LIMIT 5

@jwang thanks for confirming that. With that information, I determined that the most likely event then is that we accidentally enabled this on June 22nd as the query looks solid, as the action compact-language-links-open and context header can only occur if the user is seeing the language button in the header.

I looked into this and sure enough on the 22nd June we enabled for all anonymous users in the test group, https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/700705/5/wmf-config/InitialiseSettings.php
So the deployment agenda needs updating.
FYI @ovasileva

@jwang with this new date in mind, it looks like the data makes sense again?

Deployment date: 2021-06-22
Catch-up period : 2021-06-22 ~ 2021-06-24 Due to caching, not all users were seeing the new version. The cached articles were displaying the old version of the language switcher. The cache tends to clear completely within two days, which is why we exclude these two days in analysis

@Jdlrobson thank you for checking and confirming it. I have updated "Deployment Agenda" to

Deployment date: 2021-06-22
Catch-up period : 2021-06-22 ~ 2021-06-24 Due to caching, not all users were seeing the new version. The cached articles were displaying the old version of the language switcher. The cache tends to clear completely within two days, which is why we exclude these two days in analysis
Post-deployment: 2021-06-25 ~ 2021-07-24

FYI @ovasileva in case other documentations need an update. I will continue pre/post analysis using 2021-06-22 as the deployment date.

In T289200#7316004, @jwang wrote:

@Jdlrobson thank you for checking and confirming it. I have updated "Deployment Agenda" to

Deployment date: 2021-06-22
Catch-up period : 2021-06-22 ~ 2021-06-24 Due to caching, not all users were seeing the new version. The cached articles were displaying the old version of the language switcher. The cache tends to clear completely within two days, which is why we exclude these two days in analysis
Post-deployment: 2021-06-25 ~ 2021-07-24

FYI @ovasileva in case other documentations need an update. I will continue pre/post analysis using 2021-06-22 as the deployment date.

Thanks @jwang, @Jdlrobson! I will go ahead and update the documentation with the new dates

@ovasileva , report is posted at: https://github.com/jenniferwang-wmf/Web_language_switch/blob/master/Language_switching_logout_user_before_after_report.ipynb

Summary

• language link clicks

There was an average 61.1% decrease in total clicks on language links by logged-out users on the early adopter wikis following deployment of the new language feature. We saw a similar drop on logged-in users in the AB test. The decrease is contradicted with our hypothesis. It might be due to the lags in new feature adoption. Further investigation may be needed to identify possible reasons for this decrease.

• input language link clicks

There was an average 69.9% increase in daily average clicks on input language links by logged-out users on the early adopter wikis following deployment of the new language feature. The average is driven upwards mainly by Bengali Wikipedia (460.53%) and French Wiktionary (240.79%) and Hebrew Wikipedia (129.24%). They are wikis with low daily activities. If we exclude wikis which have less than 1 in daily average clicks, the average change is -3.038%.

Discussed with Olga about where the drop might come from. Decided to pull the data of button clicks to check whether the drop is from the first step of the new language switch. The old feature lists the language links on sidebar without a button to start with. So below comparison is an apple-to-orange comparison. But it still gives us a sense that the clicks on new button is a lot less than on the old feature on sidebar.

@jwang - we were also discussing that the drop might be resulting from the time it requires to scroll to the top of the page (in which case it might be alleviated by the new sticky header). Do you think we can look at the time to select button for before and after and see if there's any pattern?

To investigate where the drop is from, we also slice and dice the data by time-to-change and browser.

• Time to change

The distribution of the time to change on new feature is similar to old feature. But median of the time to change on new feature is slightly longer than old feature.

• Clicks by browser

The trends of clicks are similar on 4 major browsers: Chrome , Edge, Firefox, Safari. On all browsers, the clicks after deployment is much less than before.

When this feature was first deployed at Basque Wikipedia there were some (more than one) readers that thought that there wasn't any other language version of the article. This can be normal, because all Wikipedias have their links there except a few. Also, the interwiki link was pulled to the bottom of the Main Page, because no one has thought that the main page also must be redesigned and doesn't have a title. So, given that the most viewed page in every wiki is the main page, and that the feature was (practically) deleted, a decrease in clicks should be obvious.

On the other hand, the new system is worse, because it asks for one click more, and it is more difficult to open a new tab with the center button of the mouse. One thing I usually did when exploring a topic, was opening 3-4 tabs just clicking on the sidebar, so I could see the same article in some other reference languages. Now this is virtually impossible, because you have to do it one by one and if you click in the center button of the mouse you get this slider button:

So, the most practical thing is just going to another Wikipedia and using the old language selector to open it in more languages.

We can expect that people will get used to the new location, it will be easier when this feature is deployed in every wiki, but we can't expect to have more clics if the system is worse and can't be located in the Main Page.

Is there any update to the difference of clics at the Main Page at euwiki once the BUG of putting it in the bottom was solved simpy using CSS? It would be interesting to know if people is not clicking just because the feature was hidden and couldn't find it or because they don't understand the new system.