Background
The goal for IP Info is to provide patrollers the ability to get critical IP-related information in MediaWiki instead of relying on external websites to get this information. This tool will be very valuable as we move towards masking IPs. We would like to measure the usage of this tool.
What do we want to learn?
- Is the information surfaced in IP Info useful?
- Is the tool being used/accessed?
Both for the popup and the accordion.only for popup because we cannot measure accordion. Reason: Accordion is expanded or collapsed based on its previous status. Users can access IP info without clicking the accordion. Meanwhile, users visit the "User contributions" page for two possible purposes: to view the IP info, or to view the contributions. We cannot measure the accordion by the number of visits to the "User contributions" page.
- Is the tool being used/accessed?
- How often is the accordion expanded/collapsed?
- How often is the accordion opened *after* the popup was opened?
- How often is limited view shown? How often is full view shown?
- Is the information being surfaced where it is needed?
- What is the usage of the popup on the different pages where it is exposed?
- Special:Log (including action=history)
- Special:Contributions
- Special:RecentChanges
- What is the usage of the popup on the different pages where it is exposed?
- Do people understand the information being shown?
- Is the "help" for the information accessed by users?
- Who is the information being shown most useful to?
- This can help us assess if we can limit the exposure of the tool. We can measure the account age and edit count for users who are using this tool (while anonymizing their ID).
- Who opt-in / opt-out?
- Activation/Deactivation rate for the feature
- How many people see the disclaimer
- How many people accept the terms of the disclaimer
- How many people churn away
Next steps in data process (bolding indicates step we are currently on):
- PM and Data Analyst coordinate to identify experiment design, research questions, metrics and guardrails for the new feature and associated metrics.
- Data analyst creates a list of events that need to be tracked to calculate each metric. T294315
- Engineers review the event list and comment on whether we’re already tracking with existing instruments and if new instrumentation is required. New tickets created for any new instrumentation needed.
- Data analyst works with engineers to figure out what the new instrumentation should look like and where to store the events.
- Data analyst documents all of the above in the instrumentation spec
- PM signs off on the plan
Instrumentation Spec
Documentation: Instrumentation Spec