We want to set up an instrumentation/measurement plan that covers the events in the Suggested Investigations special page to give us basic visibility into how users interact with the platform.
We want to test whether such a platform is useful and desirable to CheckUsers in their day-to-day work.
The KR/Hypothesis: https://app.asana.com/1/3758245663860/project/1210702563871278/overview/1210703340891827
Design doc: https://www.figma.com/design/yO1fSBWZyMXIrADcEvlcSO/Suggested-Investigations?node-id=5540-87&p=f&t=4sp7iKOgxlMtV59u-0
Measurement plan: https://docs.google.com/document/d/19Uvgc-542fsrV_-vFnKhMRUseoN0fMQnQ_mVFyy4zmo/edit?tab=t.0
Instrumentation spec: https://docs.google.com/spreadsheets/d/1B72MHVc4Wttu9RqfC5FB_hcJP0mycWtVxhQKsv9GusY/edit?gid=0#gid=0
Questions we want to answer
- Quality of signals -> Are we generating valid investigations?
- Tool usefulness -> Do users like the workflow?
- General usage -> How do users use the tool ?
Implementation spec
User-Agent collection:
Per the instrumentation spec, the stream must be configured to opt out of collecting User-Agent information.
Contextual attributes:
The following contextual attributes need to be configured for the stream:
- agent_client_platform_family
- mediawiki_database
- performer_id
- performer_name
- performer_pageview_id
- performer_groups
- performer_edit_count
- performer_edit_count_bucket
- performer_registration_dt
Metric: The one-week retention rate of visits to the Suggested Investigations page is X%
Event to be tracked:
- User loads page
Interaction data
- action: page_load
- action_context: { is_paging_results: whether the user is on the first set of results, limit: value of the limit parameter }
Metric: Y% of CheckUsers who visit the Suggested Investigations page in a given week investigate and close at least one task
Event to be tracked:
- User loads page
- User closes a case
Interaction data
- action: case_status_change
- action_subtype: closed | invalid | open
- action_context: { case_id: numeric_task_id, signal: signal_name, number_of_users: number_of_users, has_note: whether text exists }
Metric: Proportion of cases that are closed within 48 hours out of all cases opened during a week.
Event to be tracked:
- A case is opened
Interaction data
- action: case_open
- action_subtype:
- action_context: { case_id: numeric_task_id, signal: signal_name, number_of_users: number_of_users }
Nice to have
Event to be tracked:
- A case is updated
Interaction data
- action: case_updated
- action_subtype:
- action_context: { case_id: numeric_task_id, signal: signal_name, number_of_users: number_of_users }
Acceptance criteria
- Morten w/ Madalina: Measurement plan defining metrics
- Morten: Instrumentation specification
- Engineering: Implementation of the specification
- Engineering: Instrumentation QA (pre-deployment and post-deployment)
- Morten: post-deployment data QA