Goal
To analyze the usage of fixed width toggle as part of post-deployment analysis for Vector 2022 metrics on English Wikipedia
jwang | |
Jan 23 2023, 7:46 PM |
F36824815: image.png | |
Feb 14 2023, 7:00 AM |
F36824813: image.png | |
Feb 14 2023, 7:00 AM |
F36824507: image.png | |
Feb 13 2023, 11:32 PM |
F36824497: image.png | |
Feb 13 2023, 11:32 PM |
F36824491: image.png | |
Feb 13 2023, 11:32 PM |
F36824489: image.png | |
Feb 13 2023, 11:32 PM |
To analyze the usage of fixed width toggle as part of post-deployment analysis for Vector 2022 metrics on English Wikipedia
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | jwang | T327440 Post-deployment Vector 2022 metrics analysis on English Wikipedia | |||
Resolved | jwang | T327690 Analyze usage of fixed width toggle | |||
Resolved | ovasileva | T328961 [SPIKE] QA toggle data |
Below data is not the final version. Just for discussion and code review with engineers. The data of toggle_on and toggle_off contradicts our expectation. We are expecting more toggle_off events than toggle_on events.
Analysis code: link
Below table shows toggle usage of logged-in users on English Wikipedia. The events are at 100% sample rate.
event_date | init_sessions | toggled_sessions | toggle_on_sessions | toggle_off_sessions | inits | toggle_on events | toggle_off events | session_toggle_rate (toggle_sessions/init_sessions) | toggle_on_rate (toggle_on_events/inits) |
---|---|---|---|---|---|---|---|---|---|
2023-01-16 | 9080 | 18 | 16 | 14 | 144720 | 32 | 49 | 0.1982% | 0.0221% |
2023-01-17 | 9425 | 15 | 15 | 9 | 146526 | 30 | 24 | 0.1592% | 0.0205% |
2023-01-18 | 60784 | 320 | 319 | 184 | 613935 | 734 | 484 | 0.5265% | 0.1196% |
2023-01-19 | 107934 | 429 | 424 | 257 | 1172489 | 944 | 653 | 0.3975% | 0.0805% |
2023-01-20 | 100975 | 290 | 281 | 179 | 1105554 | 650 | 489 | 0.2872% | 0.0588% |
Below table shows toggle usage of anonymous users on English Wikipedia. The events are at 1% sample rate.
event_date | init_sessions | toggled_sessions | toggle_on_sessions | toggle_off_sessions | inits | toggle_on events | toggle_off events | session_toggle_rate (toggle_sessions/init_sessions) | toggle_on_rate (toggle_on_events/inits) |
---|---|---|---|---|---|---|---|---|---|
2023-01-18 | 39516 | 45 | 45 | 6 | 41490 | 52 | 7 | 0.1139% | 0.1253% |
2023-01-19 | 272349 | 240 | 239 | 48 | 294328 | 316 | 83 | 0.0881% | 0.1074% |
2023-01-20 | 432543 | 288 | 288 | 42 | 473802 | 348 | 57 | 0.0666% | 0.0734% |
Logged-in users
Anonymous users
Tab1: Loggedin user table in datasheet shows toggle usage of logged-in users on English Wikipedia. The events are at 100% sample rate. Bot users are excluded.
Tab2: Anonymous user table in datasheet shows toggle usage of anonymous users on English Wikipedia. The events are at 1% sample rate. Bot users are excluded.
Fig1: logged-in users
Session toggle rate definition: Number of unique sessions which tried the limited width toggle feature out of number of unique initialized sessions
Fig3:
Switch to full width rate definition: Number of switches from limited width to full width out of number of sessions initialized
Fig 4
We investigated the final width option selected by each user for each session as of February 12, 2023. In some instances, the clicks were so rapid that the recorded timestamps are identical, making it impossible to determine the selected width option. As a result, we have excluded sessions where the final series of actions was recorded with the same timestamps.
Logged-in users
Tab3 shows the last action logged-in user made per session between 2023-01-18 (after deployment) and 2023-02-12.
full_width | unique sessions |
---|---|
limited-width-toggle-off (full-width -> limited width layout) | 1046 |
limited-width-toggle-on (limited width layout -> full-width) | 1823 |
Note
Anonymous users
Tab4 shows the last action anonymous user made per session between 2023-01-18 (after deployment) and 2023-02-12.
full_width | unique sessions |
---|---|
limited-width-toggle-off (full-width -> limited width layout) | 300 |
limited-width-toggle-on (limited width layout -> full-width) | 3611 |
Note
Instrumentation discussion: T328961
@ovasileva, as mentioned in T327690#8612771, we saw more switches from limited width to full width were made per pageviews on anonymous users. Below analysis looked into the switches from limited width to full width per each session which had used toggle features. Data did not show more clicks per session on anonymous user.
Will switches per session be higher on anonymous users than logged-in users, given the toggle setting is not persistent on anonymous users?
We looked into switches from limited width to full width per session, and did not see more clicks per session on anonymous users than loggedin users. The lower pageviews per session on anonymous users may be the reason. The logged-in users have a pageviews to session ratio of 10.8, significantly higher than that of anonymous users, which is 1.1. Anonymous users just read 1.1 pages per session. (data source)
Tab: Number of switches from limited width to full width per session
Loggedin Users | Anonymous users | |
---|---|---|
min | 1 | 1 |
1st quartile | 1 | 1 |
Median | 1 | 1 |
3rd quartile | 3 | 1 |
Max | 91 | 408 |
Mean | 2.586 | 1.457 |
Loggedin Users | Anonymous users | |
---|---|---|
min | 1 | 1 |
1st quartile | 1 | 1 |
Median | 1 | 1 |
3rd quartile | 2 | 1 |
Max | 26 | 408 |
Mean | 1.697 | 1.303 |
SELECT event.isAnon, event.token AS session_token, event.pageToken AS page_tokens, event.name AS full_width, meta.dt AS dt, rank() over (partition by event.token order by meta.dt ) AS rnk FROM event.desktopwebuiactionstracking WHERE year=2023 AND wiki='enwiki' AND meta.dt > '2023-01-18T15:30:00.000Z' AND TO_DATE(meta.dt) <= '2023-02-12' AND event.action='click' AND event.name IN ('limited-width-toggle-on', 'limited-width-toggle-off') -- exclude bot AND NOT useragent.is_bot -- only vector-2022 has toggle button AND event.skin='vector-2022'
Code
# Only look in swtich from default to full width df_clicks_per_session_6 <- df_session %>% filter(full_width=='limited-width-toggle-on') %>% group_by(isAnon, session_token) %>% summarize( number_clicks=n(),.groups='drop') df_clicks_per_session_6 %>% filter(isAnon=='False') %>% select(number_clicks) %>% summary() df_clicks_per_session_6 %>% filter(isAnon=='True') %>% select(number_clicks) %>% summary() # Only look in swtich from default to full width df_clicks_per_session_page2 <- df_session_page %>% filter(full_width=='limited-width-toggle-on') %>% group_by(isAnon, session_token, page_tokens) %>% summarize( number_clicks=n(),.groups='drop')
Interesting. How did this change once the toggle was made persistent for anonymous users? Does this include sessions where screens were too small to ever see the toggle?
Is it possible to experiment with different sizes of the toggle, or survey users to find out how many know the toggle exists?