Page MenuHomePhabricator

Year-end report on FY2023-24 KR WE2.1
Closed, ResolvedPublic

Description

Background

At the end of the fiscal year, we plan to have deployed a number of new customization options that adapt the default experience. We would like to perform a year-end report on KR 2.1 Ensure a quality reading experience for all users by adapting the default experience for 15% of sessions, based on the individual needs and constraints of the user.

User story

  • As a product manager, I want the ability to know what impact the features build over the course of the year had so that we can determine whether we reached our KR targets

Requirements

Report on the following based on current data:

Establish %age of non-default sessions on Vector22
% of sessions which do not use the default width option (and view the site at full width) Vector22 [INTERIM REPORTING NEEDED for sessions where viewport >= 1200px]
% of sessions which have set a non-default font size (Vector22)
% of sessions using non-default color mode (on Vector-2022)
% of sessions which do not use page previews in the Vector 2022 skin (or on desktop overall)
% of sessions which are viewing a pinned/unpinned version of the menus on the site (main menu, tools menu) Vector 22
% of sessions which have disabled media viewer

Establish %age of non-default sessions on Minerva
% of sessions using a non-standard font size (on Minerva) [INTERIM REPORTING NEEDED]
% of sessions using non-default color mode (on Minerva)

Use the above to report on:
Overall % of sessions with adapter default experience on Minerva skin
Overall % of sessions with adapter default experience on Vector22 skin (limited to screen resolutions >1200px)

Resources:
Web team supporting notes
Web Team Year-end report on KR WE2.1
Notebooks

Key details for analysis:
Deployed (Everything but darkmode is out) on June 11th on all Wikipedias

Feature event names:
event.is_page_preview_on
event.is_full_width
event.is_media_viewer_enabled
event.is_pinned
event.dark_mode_setting
event.font

Done

Analysis result: data sheet.

Details

Other Assignee
jwang

Event Timeline

ovasileva renamed this task from Year-end report on KR 2.1 to Year-end report on KR WE2.1.May 9 2024, 2:19 PM
Aklapper renamed this task from Year-end report on KR WE2.1 to Year-end report on FY2023-24 KR WE2.1.May 9 2024, 4:52 PM
mpopov updated Other Assignee, added: jwang.
mpopov added a subscriber: jwang.

Data for

  • % of sessions which do not use the default width option (and view the site at full width) Vector22 [INTERIM REPORTING NEEDED for sessions where viewport => 1200px]
  • % of sessions using a non-standard font size (on Minerva) [INTERIM REPORTING NEEDED]

...can be found in this sheets file.

Update: T349862 is impacting data reporting.
From @KSarabia-WMF: The discrepancy noting that the majority of logged-out users are using the small font size is due to a mismatch in the HTML form where the small value is labeled as "Standard" in Minerva.
In Desktop, the default and "Standard" is "1".

Hi @KSarabia-WMF, can you clarify the font values captured in event.MobileWebUIActionsTracking schema for each font option below? What's the default value? Are these values consistent across logged-in and logged-out users? The info will help me understand whether we can adjust the results using the current data.

Screenshot 2024-07-17 at 12.34.11 PM.png (1×2 px, 185 KB)

Font Options and Their Captured Values (UPDATED)
  1. Minerva (Mobile):
    • Standard (Default): small
    • Medium: regular
    • Large: large
  1. Vector (Desktop):
    • Small: 0
    • Standard (Default): 1
    • Large: 2
Default Value
  • Minerva (Mobile):
    • The default font size for logged-out users is small (Users see this as "Standard").
  • Vector (Desktop):
    • The default font size for both logged-in and logged-out users is 1 (Users see this as "Standard").
Consistency Across Logged-In and Logged-Out Users
  • Logged-In Users:
    • Font values are correctly captured based on user selection.
  • Logged-Out Users:
    • In Minerva (Mobile), the default value is small (Users see this as "Standard").
    • In Vector (Desktop), the default value is 1 (Users see this as "Standard").

Summary of Findings

  1. Font Values Captured:
    • Minerva (Mobile):
      • Standard (Default): small
      • Medium: regular
      • Large: large
    • Vector (Desktop):
      • Small: 0
      • Standard (Default): 1
      • Large: 2
  1. Default Value:
    • Minerva (Mobile): Default for logged-out users is small (Users see this as "Standard").
    • Vector (Desktop): Default for both logged-in and logged-out users is 1 (Users see this as "Standard").
  1. Consistency:
    • The captured values are consistent across both logged-in and logged-out users are currently inconsistent. A ticket is in the works to fix this.
  2. Context:
    • The issue is that we kept the legacy 14px value in vector as a small. So the "Standard" was the second biggest size. In Minerva, it was already 16px, which we wanted as the new default and didn't want anything smaller.

@jwang See the write-up above

@KSarabia-WMF , thanks for taking a look. I am confused. I did not see some of the values you listed for Minerva (Mobile) on interface and schema. Can you double check?

What I observed for Minerva (Mobile):
On interface: standard, medium, large
Values in schema: regular, large, small, 0

@ovasileva, @KSarabia-WMF
Can you provide the timeline of the deployments? Any other changes to existing features?

I've noticed a significant change in the font feature trend on desktop: the numbers of small font (0) and standard font (1) altered dramatically around June 12. It appears that the default value may have been changed around that date. If this is the case, we may need to adjust our analysis window due to caching issues.

image.png (1×1 px, 147 KB)

@jwang Sorry, you're right. I am confusing myself. standard, medium, large is what is on the interface.
For the following

Values in schema: regular, large, small, 0

where are you seeing that/what are you referencing?

where are you seeing that/what are you referencing?

The value captured in schema event.MobileWebUIActionsTracking

The query I used:

> SELECT distinct  event.font
> FROM event.MobileWebUIActionsTracking
> WHERE year=2024 and month=6 
> AND event.action='init'

@jwang I've updated the comment above with the correct values. Again, sorry for the confusion. The results of that query looks as expected.

Vector (Desktop):
The default font size for both logged-in and logged-out users is 1 (Users see this as "Standard").

@KSarabia-WMF, Can you double check the default font value of logged-in on desktop? The below graph shows more sessions with font of 0 than 1. It seems the default value is different for logged-in and logged-out.

image.png (1×1 px, 108 KB)

@KSarabia-WMF, Can you confirm whether the below instrumentation mapping table is correct for mobile Minerva users?

event.dark_mode_settingcolor theme
0 (default)light
1dark
2Automatic

Is the default same for desktop users?

Here is the analysis summary of font menu. CC @ovasileva

Methodology

The non-default font rate was defined as the ratio of unique sessions with a non-default font size to the total number of unique initialized sessions per day. We analyzed the daily average over a specified timeframe.

Mobile web (Minerva skin)

Analysis request: % of sessions using a non-standard font size (on Minerva)
Summary:

  • For logged-in users on mobile, the non-default font rate is 2.052%
  • For anonymous users on mobile, the non-default font rate is 0.015%

Compared with baseline

FeatureUser typeBaselineYear-end reportDefault value as of July 23, 2024Baseline analysis timeframeYear-end analysis time frame
Font menuLogged-in1.241%2.052%Standard font (value: small)Nov.4 ~Nov.8 , 2023June 11 ~ June 27, 2024
Font menuAnonymous0.014%0.015%Standard font (value: small)Nov.4 ~Nov.8 , 2023June 11 ~ June 27, 2024

Note:
The default value changed since our baseline analysis. In baseline analysis (T346979) , the default font is Medium ( schema value: regular). In the year-end analysis, the default font is Standard (schema value: small).

Desktop (Vector2022 skin)

Analysis request: % of sessions which have set a non-default font size (Vector22)
Deployment note: deployed by tiers. It has been fully deployed on all wikis by Jun 11 (T362148#9879986)
Summary:

  • For logged-in users on desktop, the non-default font rate is 13.29%
  • For anonymous users on desktop, the non-default font rate is 8.00%

Compared with baseline

FeatureUser typeBaselineYear-end reportDefault value as of July 23, 2024NoteBaseline analysis timeframeYear-end analysis time frame
Font menuLogged-inNA13.29%Small ( value: font=0)NAJuly 8 ~ July 21, 2024
Font menuAnonymousNA8.00%Standard (value: font=1)The default value changed from small font (0) to standard font (1) around June 12.NAJuly 8 ~ July 21, 2024

Here is the analysis result of color-theme (dark mode) and overall on Mobile web (Minerva skin). cc @ovasileva

Methodology

The non-default color-theme rate was defined as the ratio of unique sessions with a non-default color-theme setting to the total number of unique initialized sessions per day. The non-default settings include: Dark and Automatic.
The over-all non-default rate on mobile was defined as the ratio of unique sessions with a non-default setting of font size OR color theme to the total number of unique initialized sessions per day.
We analyzed the daily average over a specified timeframe.

Results

Analysis request: % of sessions using non-default color mode (on Minerva)
Summary:

  • For logged-in users on mobile, the non-default color-theme rate is 2.83%.
  • For anonymous users on mobile, the non-default color-theme rate is 0.18%.

Analysis request: Overall % of sessions with adapter non-default experience on Minerva skin
Summary:

  • For logged-in users on mobile, the overall non-default rate is 4.63%.
  • For anonymous users on mobile, the overall non-default rate is: 0.20%.
FeatureUser typeBaselineYear-end reportDefault value as of July 23, 2024Baseline analysis timeframeYear-end analysis time frame
Dark modeLogged-inNA2.83%light (0)NAJuly 24 ~ July 30, 2024
AnonymousNA0.18%light (0)NAJuly 24 ~ July 30, 2024
Overall (font and dark mode)Logged-inNA4.63%Standard font (value: small) & light theme (value: 0)NAJuly 24 ~ July 30, 2024
AnonymousNA0.20%Standard font (value: small) & light theme (value: 0)NAJuly 24 ~ July 30, 2024

Here is the analysis result of color-theme (dark mode) and overall on desktop. cc @ovasileva

Results

Analysis request: % of sessions using non-default color mode (on Vector-2022)
Summary:

  • For logged-in users, the non-default color theme rate is 7.04%
  • For anonymous users, the non-default color theme rate is 1.31%

Analysis request: Overall % of sessions with adapter default experience on Vector22 skin (limited to screen resolutions >1200px)
Summary:

  • For logged-in users, the non-default is 60.0%.
  • For anonymous users:, the non-default is 10.1%.

Please find the analysis of full width, media viewer, pin, preview and comparison with baselines at data sheet.