Page MenuHomePhabricator

[EPIC] Instrument and analyze success of user links menu
Closed, ResolvedPublic

Description

Background

This epic will contain all tasks related to evaluating the success of the new user links menu: https://www.mediawiki.org/wiki/Reading/Web/Desktop_Improvements/Features/User_menu. We will be monitoring the before and after usage of the user links on our pilot wikis following our initial deployments. While we expect to see a small drop in user links, any drop over 10% will be considered ground for immediate iteration on the feature.

Main questions:

How does usage of the links decrease or increase before and after usage of the feature. For this, we will not be performing an A/B test, but rather a before and after comparison between the current implementation (for which we should have historical data via the DesktopWebUIActionsTracking schema and compare to the current implementation. Open question for this spike - does the new version track events in a similar way?
Do these changes affect account creation - it's possible we might see an increase or decrease here. For this we would do a before/after comparison. Note: we're already tracking this in the desktop improvements superset dashboard: https://superset.wikimedia.org/superset/dashboard/153/
How do edits to these pages change once the changes are implemented? For example, we can look at edits of the sandbox or user page before/after per user

Instrumentation Spec

Please refer to the following instrumentation spec to track and document instrumentation requirements:
https://docs.google.com/spreadsheets/d/1ZvHcy6lXevrdd-t_8fT2MmHBsQK3aFdLNGSRE6IiWb8/edit#gid=0

Event Timeline

@ovasileva I've updated the task description with the instrumentation spec for the user links menu. I've filled it in with all of the metrics and status of existing or needed instrumentation identified in T284542. This can be referenced and updated as we make any needed changes to the existing instrumentation.

@Jdlrobson As you mentioned in the weekly planning meeting, it would be useful to create a glossary of the unique names for all the user menu links and document any difference between wikis. I added a "Glossary" tab to the instrumentation spec called that can be used if helpful but this would ideally be documented on maybe a README file in the schema repo or mediawiki page. Should we create a new task for this work?

@MNeisler I've added a list of all the user links we're tracking (in bold) and everything outside in case that's of interest (not bolded). Note on certain wikis there may be gadgets we need to consider but I think it would be enough to focus on the ones bolded before and after. I've added a column to specify user links that have been added as part of the project that are not present in the old version. When analyzing these and comparing to the old version we need to consider these.

For example: clicks to create account

Before change: Consider clicks to pt-createaccount
After change: Consider clicks to pt-createaccount AND p-createaccount

Does that make sense? Let me know if you have any questions or I need to provide any follow ups.

ovasileva claimed this task.