@ovasileva Thank you for creating this! We will triage it at a team meeting tomorrow (23 February).

Discussed in detail with Olga about how we can measure the effects. Added additional details to description.

added some notes on what we would need for the schema itself

Moving to Current Quarter - as this is not a Spike and the actual Analysis task assigned to Product analytics

In T245735#6217074, @Mayakp.wiki wrote:

The collapsible sidebar and its instrumentation will be deployed on July 1. Thus, tentative timeline for analysis to start will be around July 1st week.
Assigning task to Megan per 1:1 with Olga today.

A couple of notes here - I think we should probably give it at least a couple of weeks before beginning analysis. I'd say mid-July feels more realistic. Will update once a deployment date is set - right now, it depends on finishing up some of the remaining work

hey @ovasileva , can we have this task updated to include the use of VectorSkinVersion that we spoke about in our last 1:1 ?

we need the instrumentation to run on both versions of the Vector skin hence it makes sense to have skin version added to the schema (to establish a baseline and then be able to compare with users that opt-out later on).

However, if you feel this baseline and comparison is okay to be analyzed at a later time then we can maybe create a separate task for it i.e. not add scope to this task. Please let us know what you would like.

@ovasileva Here is the report on the usage of the collapsible sidebar on the test wikis. Let me know if you have any questions or suggested changes.

Below is a summary of some key observations and highlights for reference:
What is the frequency that users collapse and uncollapse the sidebar?

• Between July 22 and August 31, 2020, there were 28,745 unique sessions that included a click to collapse and/or uncollapse the sidebar. This represents only about 0.13% of all recorded sessions using the modern vector skin and 13.67% of all modern vectors sessions with a click recorded.
• Since the sidebar was set as uncollapsed by default, there are more total clicks to collapse the sidebar; however, there are roughly the same average number of clicks per session to either collapse or uncollapse. Average Clicks Per Session to Collapse: 1.37; Average Clicks Per Session to Uncollapse: 1.38.

• French Wiktionary had the lowest average sidebar clicks per session (1.87) while Persian Wikipedia had the highest (2.94).
• Logged-out users have more than twice the number of collapse events compared to uncollapse events, likely because the state of the sidebar for logged out users is not preserved.
• Most sessions overall included just one-click to either collapse or uncollapse the sidebar. Logged out users have a higher frequency of one sidebar clicks per session while logged in users have a higher frequency of two clicks per session.

What are the different interaction types with the sidebar?

• For the majority of sessions (81.88%), the sidebar was collapsed and/or uncollapsed just once. For sessions where the sidebar was collapsed only once, 34.5% sessions kept the sidebar collapsed the majority of their session time and 65.4% of sessions had the sidebar uncollapsed the majority of their session time (did not collapse it until towards the end of their session).

Sidebar Interactions Summary

[1] Clicks were more than than 1 second apart

**What is the overall difference in interaction with sidebar links before and after the change?**
Note re approach:
Since this was deployed as opt-out to all users on the test wikis, there is not a random, equal split of users between the two vector skins so a direct comparison cannot be made as there may be some bias (the legacy skin version was only available to those that explicitly opted out) and also there more users that had the default experience. However, I compared the average interactions per session for each skin type to control for the different number of users and confirm if there were any significant differences. Note: Data in these comparisons may still be impacted by self-selection bias between the two groups. I also reviewed data from a two-week period before and after each test wiki's deployment date to confirm if there were any significant changes.

• The legacy skin has a slightly higher number of total sidebar link clicks per session (4.68) compared to modern (3.63). This is also true on a per wiki basis except for on French Wiktionary and Portuguese Wikiversity, which have a slightly higher average number of total sidebar link clicks on the modern skin compared to legacy.
• Clicks to the donate link do not appear to be impacted by the new skin version. There is a slightly higher number of average clicks per session on the modern skin (1.11) compared to the legacy skin (1.06).
• There was a decrease (-9.15%) in the overall total number of sidebar link interactions comparing the two weeks prior and two week following the deployment of the collapsible sidebar across all test wikis; however, this percentage is likely inflated due to seasonality effects (previous year's data was not available using webrequest to complete a year over year analysis).

Please see the report and codebase for further breakdowns and details.