Page MenuHomePhabricator

Analyze effects of collapsible sidebar
Closed, ResolvedPublic

Description

Background

We would like to study the ways in which the collapsible sidebar is affecting user behavior. This task is created to determine the instrumentation required for us to do so.

Acceptance criteria

  • Answer the following questions
  • What is the frequency that users collapse and uncollapse the sidebar
  • What is the difference in interaction with sidebar links between people with the sidebar collapsed (the majority of the time) versus uncollapsed (the majority of the time). i.e. Once the sidebar is collapsed, do users open it when they need to, or do they just ignore it altogether?
  • What is the overall difference in interaction with sidebar links before and after the change
  • In particular, what is the overall difference in interaction with the donate link?
Schema criteria
  • state of sidebar
  • session id
  • user status (logged-in, logged-out)
  • edit count bucket
  • timestamp
  • clicks to donation link
  • skin version (From task T250282 : Instrumentation should be run on both the new version and old version of the skin on a given wiki)
Additional Context around the analysis

What is the goal of this analysis? why do yo want to measure the effects of the collapsible sidebar? This would be helpful to understand this in order to determine if we need to consider more or less in the effects do we need to measure'.

  • Usage for sidebar is low and each link is not instrumented. Web team is not looking for a Success metric, the simply want to understand the effects and want to see how ppl are using the sidebar.

except Donate link: for fundraising

What is the frequency per session that users collapse and uncollapse the sidebar (logged-in, logged-out, edit count bucket, wiki)
What is the default state of the sidebar?

  • default uncollapsed until language switcher deployment. after - default collapsed for anon, uncollapsed for logged-in. Logged-in - preserve your choice across sessions

Do you mean, how many people collapse an uncollapsed sidebar? how many people discover that the sidebar is collapsible?

  • Yes

Will need instrumentation to track the 'collapsing' action;

  • Interaction with the button per session.

Will need new Schema that can be used for this and other features that will be rolled out as a part of DIP
Fyi, Privacy issue to instrument each link in sidebar. will need legal consultation if we ever think about it. Not required now.

What is the difference in interaction with sidebar links between people with the sidebar collapsed (the majority of the time) versus uncollapsed (the majority of the time). i.e. Once the sidebar is collapsed, do users open it when they need to, or do they just ignore it altogether? (add session id, timestamp to new schema)
Define 'majority of time' i.e. how much time?

  • given an average session length, do ppl have it open or closed for 50% of the time of their avg session length.

Do you mean 'Of all the people that collapse the sidebar, how many people reopen it i.e. uncollapse it again?

  • Open or closed for time per session; recording the state and having it per user
  • which of these , open or close is more popular ?
  • how does that behavior present itself

**What is the overall difference in interaction with sidebar links before and after the change

  • (rerun Megan's report : sidebar and lang links)

Define interaction: is it collapse-uncollapse; or click on a sidebar menu

  • traffic to each sidebar link using webrequest data

Will need to instrument the lang links when it is deployed, later

In particular, what is the overall difference in interaction with the donate link?
Will need to instrument 'Donate' link.
Fundraising knows current interaction details. we will check the numbers 'after', once instrumented, in our analysis

Timeline for Analysis:
Start Analysis around Aug 17th and complete in early September

Event Timeline

ovasileva renamed this task from [SPIKEPMeasure effects of collapsible sidebar to [SPIKE] Measure effects of collapsible sidebar.Feb 20 2020, 12:35 PM
ovasileva triaged this task as High priority.
ovasileva updated the task description. (Show Details)
ovasileva added a subscriber: Mayakp.wiki.

@Mayakp.wiki - creating a task for the instrumentation of the sidebar. Please let me know if I've left anything out.

ovasileva updated the task description. (Show Details)

@ovasileva Thank you for creating this! We will triage it at a team meeting tomorrow (23 February).

MBinder_WMF changed the subtype of this task from "Task" to "Spike".Mar 3 2020, 6:13 PM

Discussed in detail with Olga about how we can measure the effects. Added additional details to description.

added some notes on what we would need for the schema itself

ovasileva renamed this task from [SPIKE] Measure effects of collapsible sidebar to Measure effects of collapsible sidebar.Apr 15 2020, 2:18 PM
Mayakp.wiki changed the subtype of this task from "Spike" to "Task".May 4 2020, 6:30 PM
Mayakp.wiki moved this task from Epics to Current Quarter on the Product-Analytics board.
Mayakp.wiki removed a project: Epic.

Moving to Current Quarter - as this is not a Spike and the actual Analysis task assigned to Product analytics

ovasileva renamed this task from Measure effects of collapsible sidebar to Analyze effects of collapsible sidebar.Jun 11 2020, 7:33 PM
Mayakp.wiki moved this task from Current Quarter to Upcoming Quarter on the Product-Analytics board.

The collapsible sidebar and its instrumentation will be deployed on July 1. Thus, tentative timeline for analysis to start will be around July 1st week.
Assigning task to Megan per 1:1 with Olga today.

The collapsible sidebar and its instrumentation will be deployed on July 1. Thus, tentative timeline for analysis to start will be around July 1st week.
Assigning task to Megan per 1:1 with Olga today.

A couple of notes here - I think we should probably give it at least a couple of weeks before beginning analysis. I'd say mid-July feels more realistic. Will update once a deployment date is set - right now, it depends on finishing up some of the remaining work

MNeisler lowered the priority of this task from High to Medium.Jun 22 2020, 4:35 PM

hey @ovasileva , can we have this task updated to include the use of VectorSkinVersion that we spoke about in our last 1:1 ?

we need the instrumentation to run on both versions of the Vector skin hence it makes sense to have skin version added to the schema (to establish a baseline and then be able to compare with users that opt-out later on).

However, if you feel this baseline and comparison is okay to be analyzed at a later time then we can maybe create a separate task for it i.e. not add scope to this task. Please let us know what you would like.

ovasileva raised the priority of this task from Medium to High.Sep 8 2020, 1:43 PM

@ovasileva Here is the report on the usage of the collapsible sidebar on the test wikis. Let me know if you have any questions or suggested changes.

Below is a summary of some key observations and highlights for reference:
What is the frequency that users collapse and uncollapse the sidebar?

  • Between July 22 and August 31, 2020, there were 28,745 unique sessions that included a click to collapse and/or uncollapse the sidebar. This represents only about 0.13% of all recorded sessions using the modern vector skin and 13.67% of all modern vectors sessions with a click recorded.
  • Since the sidebar was set as uncollapsed by default, there are more total clicks to collapse the sidebar; however, there are roughly the same average number of clicks per session to either collapse or uncollapse. Average Clicks Per Session to Collapse: 1.37; Average Clicks Per Session to Uncollapse: 1.38.

  • French Wiktionary had the lowest average sidebar clicks per session (1.87) while Persian Wikipedia had the highest (2.94).
  • Logged-out users have more than twice the number of collapse events compared to uncollapse events, likely because the state of the sidebar for logged out users is not preserved.
  • Most sessions overall included just one-click to either collapse or uncollapse the sidebar. Logged out users have a higher frequency of one sidebar clicks per session while logged in users have a higher frequency of two clicks per session.

What are the different interaction types with the sidebar?

  • For the majority of sessions (81.88%), the sidebar was collapsed and/or uncollapsed just once. For sessions where the sidebar was collapsed only once, 34.5% sessions kept the sidebar collapsed the majority of their session time and 65.4% of sessions had the sidebar uncollapsed the majority of their session time (did not collapse it until towards the end of their session).

Sidebar Interactions Summary

Session BehaviorNum_sessionsPercent of sessions with click to collapse and/or uncollapse
Collapse but don't uncollapse1347546.91%
Uncollapse but don't collapse10433.63%
Collapse once and then uncollapse once [1]2420.84%
Uncollapse once and then collapse once [1]260.09%
Both Collapse and Uncollapse under 1 second873330.40%
Collapse and/or Uncollapse Multiple times within a session520418.12%

[1] Clicks were more than than 1 second apart

**What is the overall difference in interaction with sidebar links before and after the change?**
Note re approach:
Since this was deployed as opt-out to all users on the test wikis, there is not a random, equal split of users between the two vector skins so a direct comparison cannot be made as there may be some bias (the legacy skin version was only available to those that explicitly opted out) and also there more users that had the default experience. However, I compared the average interactions per session for each skin type to control for the different number of users and confirm if there were any significant differences. Note: Data in these comparisons may still be impacted by self-selection bias between the two groups. I also reviewed data from a two-week period before and after each test wiki's deployment date to confirm if there were any significant changes.

  • The legacy skin has a slightly higher number of total sidebar link clicks per session (4.68) compared to modern (3.63). This is also true on a per wiki basis except for on French Wiktionary and Portuguese Wikiversity, which have a slightly higher average number of total sidebar link clicks on the modern skin compared to legacy.
  • Clicks to the donate link do not appear to be impacted by the new skin version. There is a slightly higher number of average clicks per session on the modern skin (1.11) compared to the legacy skin (1.06).
  • There was a decrease (-9.15%) in the overall total number of sidebar link interactions comparing the two weeks prior and two week following the deployment of the collapsible sidebar across all test wikis; however, this percentage is likely inflated due to seasonality effects (previous year's data was not available using webrequest to complete a year over year analysis).

Please see the report and codebase for further breakdowns and details.

Report now published on-wiki. Resolving this!