Page MenuHomePhabricator

Analyze usage of fixed width toggle
Closed, ResolvedPublic

Assigned To
Authored By
jwang
Jan 23 2023, 7:46 PM
Referenced Files
F36824815: image.png
Feb 14 2023, 7:00 AM
F36824813: image.png
Feb 14 2023, 7:00 AM
F36824507: image.png
Feb 13 2023, 11:32 PM
F36824497: image.png
Feb 13 2023, 11:32 PM
F36824491: image.png
Feb 13 2023, 11:32 PM
F36824489: image.png
Feb 13 2023, 11:32 PM

Description

Goal

To analyze the usage of fixed width toggle as part of post-deployment analysis for Vector 2022 metrics on English Wikipedia

Event Timeline

Below data is not the final version. Just for discussion and code review with engineers. The data of toggle_on and toggle_off contradicts our expectation. We are expecting more toggle_off events than toggle_on events.
Analysis code: link

Below table shows toggle usage of logged-in users on English Wikipedia. The events are at 100% sample rate.

event_dateinit_sessionstoggled_sessionstoggle_on_sessionstoggle_off_sessionsinitstoggle_on eventstoggle_off eventssession_toggle_rate (toggle_sessions/init_sessions)toggle_on_rate (toggle_on_events/inits)
2023-01-16908018161414472032490.1982%0.0221%
2023-01-1794251515914652630240.1592%0.0205%
2023-01-18607843203191846139357344840.5265%0.1196%
2023-01-1910793442942425711724899446530.3975%0.0805%
2023-01-2010097529028117911055546504890.2872%0.0588%

Below table shows toggle usage of anonymous users on English Wikipedia. The events are at 1% sample rate.

event_dateinit_sessionstoggled_sessionstoggle_on_sessionstoggle_off_sessionsinitstoggle_on eventstoggle_off eventssession_toggle_rate (toggle_sessions/init_sessions)toggle_on_rate (toggle_on_events/inits)
2023-01-183951645456414905270.1139%0.1253%
2023-01-1927234924023948294328316830.0881%0.1074%
2023-01-2043254328828842473802348570.0666%0.0734%
jwang triaged this task as High priority.Jan 31 2023, 6:10 PM
jwang moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

Summary

Logged-in users

  • Between 2023-01-18 and 2023-02-11, 0.17% of unique sessions tried toggle feature. 3.5 switches from limited width to full width were made per every 10,000 pageviews. (See Tab1 below)
  • 36.5% of logged-in user sessions opted for the limited width (default) option after exploring the full width option. (See Tab3 below)
  • During the first 4 days after deployment, there was a high utilization of the toggle feature, largely attributed to the novelty effect. One week following deployment, the daily number of switches to full width remained stable around 200. (See Fig1)

Anonymous users

  • Between 2023-01-18 and 2023-02-11, 0.04% of unique sessions tried toggle feature. 4.4 switches from limit width to full width were made per every 10,000 pageviews. (See Tab2 below)
  • 7.7% of anonymous user sessions opted for the limit width (default) option after exploring the full width option, much lower than loggedin users. (See Tab4 below). Note: Toggle setting is not persistent for anonymous users. Users may have to set the toggle option every time a new page loads, or may return to default without any clicks. And it's also possible that they may stop using the toggle feature once they realize it won't stay persistent.

Data

Data sheet

Tab1: Loggedin user table in datasheet shows toggle usage of logged-in users on English Wikipedia. The events are at 100% sample rate. Bot users are excluded.
Tab2: Anonymous user table in datasheet shows toggle usage of anonymous users on English Wikipedia. The events are at 1% sample rate. Bot users are excluded.

Daily trend of number of toggles

Fig1: logged-in users

image.png (1×1 px, 225 KB)

Note
For logged-in users, during the first 4 days after deployment, there was a high utilization of the toggle feature.
Fig2: anonymous users
image.png (1×1 px, 226 KB)

Note
For anonymous users, there was a high utilization of the toggle feature on 2023-02-08.

Daily trend of toggle rate

Session toggle rate definition: Number of unique sessions which tried the limited width toggle feature out of number of unique initialized sessions
Fig3:

image.png (1×1 px, 211 KB)

Daily trend of rate of Sessions Switching to Full Width on English Wikipedia

Switch to full width rate definition: Number of switches from limited width to full width out of number of sessions initialized
Fig 4

image.png (1×1 px, 244 KB)

Which width option the users chose to stay with?

We investigated the final width option selected by each user for each session as of February 12, 2023. In some instances, the clicks were so rapid that the recorded timestamps are identical, making it impossible to determine the selected width option. As a result, we have excluded sessions where the final series of actions was recorded with the same timestamps.

Logged-in users
Tab3 shows the last action logged-in user made per session between 2023-01-18 (after deployment) and 2023-02-12.

full_widthunique sessions
limited-width-toggle-off (full-width -> limited width layout)1046
limited-width-toggle-on (limited width layout -> full-width)1823

Note

  • 36.5% (1046/(1823+1046)) of logged-in user sessions chose limited width after explored full width.
  • 635 sessions are excluded from analysis.

Anonymous users

Tab4 shows the last action anonymous user made per session between 2023-01-18 (after deployment) and 2023-02-12.

full_widthunique sessions
limited-width-toggle-off (full-width -> limited width layout)300
limited-width-toggle-on (limited width layout -> full-width)3611

Note

  • 7.7% (300/(300+3611)) of anonymous user sessions chose limited width after explored full width.
  • 187 sessions are excluded from analysis.

Instrumentation discussion: T328961

@ovasileva, as mentioned in T327690#8612771, we saw more switches from limited width to full width were made per pageviews on anonymous users. Below analysis looked into the switches from limited width to full width per each session which had used toggle features. Data did not show more clicks per session on anonymous user.

Investigation for curiosity

Will switches per session be higher on anonymous users than logged-in users, given the toggle setting is not persistent on anonymous users?

We looked into switches from limited width to full width per session, and did not see more clicks per session on anonymous users than loggedin users. The lower pageviews per session on anonymous users may be the reason. The logged-in users have a pageviews to session ratio of 10.8, significantly higher than that of anonymous users, which is 1.1. Anonymous users just read 1.1 pages per session. (data source)
Tab: Number of switches from limited width to full width per session

Loggedin UsersAnonymous users
min11
1st quartile11
Median11
3rd quartile31
Max91408
Mean2.5861.457

image.png (1×1 px, 75 KB)

We also looked into switches from limited width to full width per session and page. The clicks per session on anonymous users is close to that of loggedin users on average, but still lower.
Tab: Number of switches from limited width to full width per session and page

Loggedin UsersAnonymous users
min11
1st quartile11
Median11
3rd quartile21
Max26408
Mean1.6971.303

image.png (1×1 px, 75 KB)

Post code here for review and discussion.
Query

SELECT event.isAnon, event.token AS session_token, event.pageToken AS page_tokens,
event.name AS full_width,
meta.dt AS dt,
rank() over (partition by event.token order by meta.dt ) AS rnk
FROM event.desktopwebuiactionstracking
WHERE year=2023
AND wiki='enwiki'
AND meta.dt > '2023-01-18T15:30:00.000Z' 
AND TO_DATE(meta.dt) <= '2023-02-12'
AND event.action='click'
AND event.name IN ('limited-width-toggle-on', 'limited-width-toggle-off')
-- exclude bot
AND NOT useragent.is_bot
-- only vector-2022 has toggle button
AND event.skin='vector-2022'

Code

# Only look in swtich from default to full width
df_clicks_per_session_6 <- df_session %>%
   filter(full_width=='limited-width-toggle-on') %>%
   group_by(isAnon, session_token) %>%
   summarize( number_clicks=n(),.groups='drop')

df_clicks_per_session_6 %>%
   filter(isAnon=='False') %>%
   select(number_clicks) %>%
   summary()

df_clicks_per_session_6 %>%
   filter(isAnon=='True') %>%
   select(number_clicks) %>%
   summary()

# Only look in swtich from default to full width
df_clicks_per_session_page2 <- df_session_page %>%
    filter(full_width=='limited-width-toggle-on') %>%
    group_by(isAnon, session_token, page_tokens) %>%
    summarize( number_clicks=n(),.groups='drop')

Interesting. How did this change once the toggle was made persistent for anonymous users? Does this include sessions where screens were too small to ever see the toggle?

Is it possible to experiment with different sizes of the toggle, or survey users to find out how many know the toggle exists?