Page MenuHomePhabricator

IP Info: Explore Instrumentation
Closed, ResolvedPublic

Description

Background

The goal for IP Info is to provide patrollers the ability to get critical IP-related information in MediaWiki instead of relying on external websites to get this information. This tool will be very valuable as we move towards masking IPs. We would like to measure the usage of this tool.

What do we want to learn?
  1. Is the information surfaced in IP Info useful?
    • Is the tool being used/accessed? Both for the popup and the accordion. only for popup because we cannot measure accordion. Reason: Accordion is expanded or collapsed based on its previous status. Users can access IP info without clicking the accordion. Meanwhile, users visit the "User contributions" page for two possible purposes: to view the IP info, or to view the contributions. We cannot measure the accordion by the number of visits to the "User contributions" page.
    • How often is the accordion expanded/collapsed?
    • How often is the accordion opened *after* the popup was opened?
    • How often is limited view shown? How often is full view shown?
  1. Is the information being surfaced where it is needed?
    • What is the usage of the popup on the different pages where it is exposed?
      • Special:Log (including action=history)
      • Special:Contributions
      • Special:RecentChanges
  2. Do people understand the information being shown?
    • Is the "help" for the information accessed by users?
  3. Who is the information being shown most useful to?
    • This can help us assess if we can limit the exposure of the tool. We can measure the account age and edit count for users who are using this tool (while anonymizing their ID).
  4. Who opt-in / opt-out?
    • Activation/Deactivation rate for the feature
    • How many people see the disclaimer
    • How many people accept the terms of the disclaimer
    • How many people churn away
Next steps in data process (bolding indicates step we are currently on):
  • PM and Data Analyst coordinate to identify experiment design, research questions, metrics and guardrails for the new feature and associated metrics.
  • Data analyst creates a list of events that need to be tracked to calculate each metric. T294315
  • Engineers review the event list and comment on whether we’re already tracking with existing instruments and if new instrumentation is required. New tickets created for any new instrumentation needed.
  • Data analyst works with engineers to figure out what the new instrumentation should look like and where to store the events.
  • Data analyst documents all of the above in the instrumentation spec
  • PM signs off on the plan
Instrumentation Spec

Documentation: Instrumentation Spec

Related Objects

Event Timeline

Niharika triaged this task as Medium priority.Oct 7 2021, 11:35 PM
Niharika created this task.
jwang updated the task description. (Show Details)
jwang updated the task description. (Show Details)

Hi @phuedx could you suggest where to store "event of disclaimer shown"? Ideally it's better to be in the same table of opt-in/out events. But I feel it doesn't fit the purpose of PrefUpdate schema where we plan to store opt-in/out events .

Hi @Niharika do we need to capture the line of history where the IPinfo click happens? I didn't see the current questions need such info to answer. But want to check with you, in case some product prototype idea in your mind need such info to analyze. We can instrument at the same time.

jwang renamed this task from IP Info: Instrumentation to IP Info: Explore Instrumentation.Nov 5 2021, 6:06 AM

I have updated instrumentation spec on :

  1. 'event' tab. Proposed field name and value in column E. A few question for discussion in Note column.
  2. 'new in schema' tab. It documents the proposed new schema and new field value for existing schema.

Feel free to suggest or propose alternatives. Thanks!

Hi @phuedx could you suggest where to store "event of disclaimer shown"? Ideally it's better to be in the same table of opt-in/out events. But I feel it doesn't fit the purpose of PrefUpdate schema where we plan to store opt-in/out events .

After discussion with @phuedex, we decided to store the events of disclaimer shown and IPInfoTool opt-in/out in IPInfoTool schema instead of PrefUpdate for two reasons:

  1. the PrefUpdate schema is not a place to store "event of disclaimer shown".
  2. In order to answer the question about enrollment funnel, we need to analyze the initial events "disclaimer shown" and the consequence events "opt-in" together. Storing them in the same table is easier for analysis.

I updated the instrumentation spec event tab and new_in_schema tab accordingly.

Niharika claimed this task.

@phuedx I think this is ready to create tickets based on the instrumentation spec.