Both skin preference and global preference reflect the status as of the data collection date, which is September 15, 2023.

@kostajh, Please see my answers below

@sgrabarczuk , thanks. I have updated the Acceptance Criteria with the full names of the wikis.

@ovasileva , I could not find wiki with language_code='nb', may I know which language wiki you are referring to?

Delivery

• Propose "Event to be tracked" in the instrumentation spec based on PM's question list.
• Propose schema field name and value for each event, and documented in instrumentation spec.
• Filled in the schemas and field names to store the events based on the discussion with engineers and PM.

Delivery

• Propose "Event to be tracked" in the instrumentation spec based on PM's question list.
• Propose schema field name and value for each event, and documented in instrumentation spec .

@ovasileva, , in order to calculate the skin stats, can you confirm whether the default skins are correct for the listed wikis?

Please create another ticket when we start to log related activities.

@KSarabia-WMF has done some investigation(T340493). Marked this ticket as resolved.

@ovasileva , Analysis reports are finalized and published at github. (see in Description). This ticket is ready for sign-off.

Below analysis focused on the top 10 browser families.

Distribution

Below figure shows the distribution of the reader sessions by top 10 browsers. Majority of readers use Chrome browser.

Pageviews

When broken down by browser family, it was observed that the treatment group exhibited an decrease in page views per session in all mobile browsers. It also decreased on Edge and Opera.

Edits

Edits per session decreased on Chrome Mobile, Edge, Mobile Safari, Opera, Safari and Samsung Internet.
Edits per session increased on Chrome Mobile iOS, Firefox.

• Have verified editattemptstep schema that skin and is_bot are captured in event field. But is_anon is NULL.

@ovasileva , Here are the updates for the viewport size analysis

Here is the summary of the analysis results for three curiosity questions about ToC usage, page tools usage, and scrolling.

Here is the update of edit analysis by measuring clicks on the edit button.

@KSarabia-WMF , thank you for the investigation.

Hi @Jdlrobson , I will cross check with desktop click tracking. Is the click on edit button logged as event.name='ca-edit' in DesktopWebUIActionsTracking ?

@olga, Here is a summary of the initial results and key findings of the AB test analysis for pageviews and edits for review. Please let me know if you have any questions.

Methodology

We reviewed AB test data recorded from 02 June 2023 through 19 June 2023 for this analysis. Data was limited to pageviews and edits completed by desktop logged-in users who were selected in the AB test.

• 466 sessions are from bot users, should be exclude in analysis.

I looked into users who have an empty up_value for up_property vector-limited-width and vector-limited-width-local-exception in mariadb on enwiki. Their user_touched timestamp in user table are within [20230202204750, 20230627212137 ]. It means they all had either logged in (not just mere visiting using an existing session), or modified user settings, or got promoted into new user groups between 2023-02-02 and 2023-06-27. [1] They likely have recently modified limited-width preference, resulting in an updated user_touched timestamp.

Mark it as resolved as all sub-tickets are resolved.

@Jdlrobson, yes it can be resolved. The analytics ticket has not been created yet.

@KSarabia-WMF, Thanks for looking into it.
Not all properties in user_properties are snapshotted in wmf_raw.mediawiki_user_properties. vector-limited-width and vector-limited-width-local-exception are not snapshotted.

The ab test events are available in mediawiki_web_ab_test_enrollment with experiment name skin-vector-zebra-experiment now.

@KSarabia-WMF, Is it possible to prolong the AB test by an additional 3 or 4 days? This way, we would obtain a full two weeks of data once the edit button issue is resolved.

Mark as resolved as the instrumentation spec has been finalized.
AB test launch will be tracked in T335972.
Adding skin key for mediawiki_web_ab_test_enrollment schema will be tracked in T335309.

Below graph shows the trend of special mute usage across wikis.

Have verified that skin field is available in mediawiki_web_ab_test_enrollment schema. The skin name is captured as expected.

@KSarabia-WMF, here is a remaining question from analysis. What the empty up_value fields in user_properties schema are representing for when you have time? Could you take a look when you have time?

• ✅ New field values have been logged.

The new event values, toc-heading.toc-pinned-disabled and toc-heading.toc-pinned-enabled have been captured since 2023-04-13.

• ✅ The links on page tools have click events logged as expected

The events of following links are confirmed to be captured: What links here, Related changes, Special pages, Page information, Cite this page, Wikidata item, Who Wrote That, Delete, Move, Change protection, Download as PDF, Printable version.
The events of clicks on Who Wrote That are not logged. It's expected.

Hi @Edtadros , can you show me what's "floaty hamburger TOC"? Originally, I thought it was the scenario when the ToC at narrow width. But from your QA summary, it seems not. Thank you!

Confirmed with Web team that Czech Wikipedia can be removed from the analysis wiki list as the vector-2022 is the default skin on this wiki.

• On the wikis where vector is the default skin, vector2022 gains more shares among editors who are more active. One possible reason is that the active editors have more chances to be exposed to the new skin version and switch over.