Page MenuHomePhabricator

Add a link: instrumentation
Closed, ResolvedPublic

Description

We have defined our research questions, measurements, experiment plan, and leading indicators in the "Add a Link - Measurement Specification". This task contains the full set of instrumentation needs, which are also copied into "instrumentation" sections on the relevant frontend tasks.


This instrumentation will exist both on the desktop and mobile versions of the feature. For users who have the Homepage enabled, we want to capture their user ID along with the information below whenever they interact with an “add a link” task.

For all measurements, we will want to:

  • Distinguish by wiki.
  • Analyze how much time elapses between each action.
  • Know whether each action was taken from desktop or mobile.
  • Know which article the user is on (through page title and page id).
  • Include timestamps with millisecond precision.

Existing instrumentation: T278119: Instrumentation: Verify existing workflows (help panel, post-edit dialog, HomepageModule)
Onboarding: T278111: Instrumentation: Onboarding
No suggestions available: T278112: Instrumentation: dialog for no suggestions available
Suggestions mode: T278114: Instrumentation: Suggestions mode
Edit mode toggle: T278115: Instrumentation: Edit mode toggle in post-release backlog
Link inspector: T278116: Instrumentation: Link inspector
Rejection dialog: T278117: Instrumentation: Rejection dialog
Edit summary: T278118: Instrumentation: Edit summary

Related Objects

Event Timeline

MMiller_WMF renamed this task from Add a Link: instrumentation to Add a link: instrumentation.Mar 13 2021, 2:04 AM
MMiller_WMF removed MMiller_WMF as the assignee of this task.
MMiller_WMF edited projects, added Growth-Team (Current Sprint); removed Growth-Team, Epic.
MMiller_WMF updated the task description. (Show Details)
MMiller_WMF added subscribers: mewoph, mepps, Rileych.
kostajh moved this task from Incoming to Epics In Progress on the Growth-Team (Current Sprint) board.

Per discussion with other engineers, I'm making this an epic and splitting out the individual parts into subtasks that can be worked on separate from their user-facing implementation; that will also allow for separate development/QA processes for these instrumentation tasks.

I'll leave links to the subtasks in the task description and reference the new tasks from their user-facing implementation tasks as well.

kostajh updated the task description. (Show Details)
kostajh updated the task description. (Show Details)
kostajh updated the task description. (Show Details)
kostajh triaged this task as Medium priority.May 4 2021, 1:45 PM

Should we do something about instrumentation not owned by us? E.g. EditAttemptStep gathers various data about the edit, which won't be comparable to the data it normally gathers (and we oversample it, so it's probably a nontrivial fraction of all EditAttemptStep data). Is there any concern of confusing other teams' analytics?

Should we do something about instrumentation not owned by us? E.g. EditAttemptStep gathers various data about the edit, which won't be comparable to the data it normally gathers (and we oversample it, so it's probably a nontrivial fraction of all EditAttemptStep data). Is there any concern of confusing other teams' analytics?

Not sure. @nettrom_WMF / @Ottomata do you have any concerns about this?

I think Morten will have to answer; I don't know much about the actual instrumentation semantics :)

Should we do something about instrumentation not owned by us? E.g. EditAttemptStep gathers various data about the edit, which won't be comparable to the data it normally gathers (and we oversample it, so it's probably a nontrivial fraction of all EditAttemptStep data). Is there any concern of confusing other teams' analytics?

ping @MMiller_WMF @nettrom_WMF about this question

@kostajh @Tgr : From what I know, the way we use EditAttemptStep means that we set the is_oversample field to true whenever we're logging a session that wouldn't normally be logged. Also as far as I know other people who work with that data know about this and will use that field to exclude oversampled sessions, but I'll mention it in our team meeting tomorrow just in case.

kostajh claimed this task.

@kostajh @Tgr : From what I know, the way we use EditAttemptStep means that we set the is_oversample field to true whenever we're logging a session that wouldn't normally be logged. Also as far as I know other people who work with that data know about this and will use that field to exclude oversampled sessions, but I'll mention it in our team meeting tomorrow just in case.

Thanks, @nettrom_WMF. I'm marking this task as resolved for now, we can re-open or making follow-up tasks as needed.