Page MenuHomePhabricator

Help panel: instrumentation for search
Closed, ResolvedPublic

Description

When we build the ability to search within the help panel (T209301), we will need to explicitly instrument the search interactions. This instrumentation will not be automatically covered by T206719, and will probably need to be designed and implemented separately.

We need to decide exactly what interactions to instrument, and then modify the help panel instrumentation document to reflect this additional work. Then we'll step through the Instrumentation DACI to get all the right approvals.

Some things we will likely want to record are:

  • Search terms people type in.
  • Search results that are selected.

Event Timeline

JTannerWMF moved this task from Inbox to FY 2019-20 on the Growth-Team board.Dec 5 2018, 4:04 PM
JTannerWMF moved this task from FY 2019-20 to Q2 2019-20 on the Growth-Team board.Jan 2 2019, 6:02 PM

The search feature is now built so we can start thinking about this.

I don't know if logging the exact search terms is going to be acceptable.

As a first step we could log a search action with how many results are returned as well as a link-click with something like "search-result-2" (referring to the second result on the list). These don't seem too controversial and would already tell us if the feature is being used at all.

RHo added a comment.Jan 11 2019, 6:52 PM

The search feature is now built so we can start thinking about this.
I don't know if logging the exact search terms is going to be acceptable.
As a first step we could log a search action with how many results are returned as well as a link-click with something like "search-result-2" (referring to the second result on the list). These don't seem too controversial and would already tell us if the feature is being used at all.

+1 - can we consider splitting the instrumentation task to first just track usage of search at all before considering whether logging specific terms is feasible? I.e., can we start with logging the following:

  • # times search is tapped overall
  • # times search is tapped per session as well as overall (ie., are people searching multiple times in a single session?)
  • # times someone clicks on a search result (without tracking specific links)
  • Activity in the help panel before and after search is used (e.g., to help answer how many people go on to ask a question after searching and presumably failing to find a result)

@SBisson -- would we be able to log these simple things (whether search is clicked into and used) in the existing HelpPanel schema? Or would we be making a new schema?

@SBisson -- would we be able to log these simple things (whether search is clicked into and used) in the existing HelpPanel schema? Or would we be making a new schema?

It fits in the current schema. We just have to update it to accept a few more action types. It's minor.

JTannerWMF edited projects, added Growth-Team (Current Sprint); removed Growth-Team.
MMiller_WMF updated the task description. (Show Details)Jan 23 2019, 12:19 AM

Next step: we are going to talk with the Search team to see how they approach this. Then we're going to come up with an approach for our purposes that fits our privacy commitments.

Update: we have asked for privacy advice on whether a simple approach to instrumentation would work, before we even have that conversation with the Search team. If the simple approach looks good, I will make a separate task for the "simple" and "advanced" versions.

Change 488935 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@master] Help panel: search instrumentation

https://gerrit.wikimedia.org/r/488935

SBisson claimed this task.Feb 7 2019, 3:00 PM

The patch above adds a new action type: 'search'. Its action_data contains the exact search query being used and the number of results returned. This is capped to 10 by default in the API so this number is likely to always or often be 10 but if some exotic queries return few results than that, we'll be able to spot it.

The search action was also added to the schema.

Also, search results click are now tracked as 'link-click' events where the link-id is search-result-N where N is the index of each result starting with 1.

@nettrom_WMF Please let me know if this works for you or if different data or events should be logged.

@SBisson -- @nettrom_WMF and I discussed, and developed specific instrumentation criteria for the basic version of instrumenting search.

High level research question: How useful is search as a feature?

More specifically, we want to answer these questions:

  • How often do people click into the search bar?
  • How often do they type something?
  • How many characters do they type?
  • How many results are returned?
  • Do they click any of the links returned?
  • Are higher results more likely to be clicked?
  • How often do users run multiple searches?
  • Do users who search for things also ask questions or click other links?

This part is up to you, but here's how we were imagining the instrumentation would change:

  • Log an event when a user clicks into the search bar.
  • Log an event hen a user clicks the back button to get out of the search context.
  • Log an event when a search is run, storing how many characters are in the search string, and the number of results -- not the actual search string.
  • Log an event when a user clicks on a search result, storing the ranking of the link they clicked, but not the specific link.

This part is up to you, but here's how we were imagining the instrumentation would change:

  • Log an event when a user clicks into the search bar.

This will be an action of type 'search-focus'. For technically reasons this one is a little trigger happy. It fires a little more than we would expect. It is possible to identify the extra events and filter them out if the exact count per session is going to be looked at.

  • Log an event hen a user clicks the back button to get out of the search context.

This will be action: 'back-home' with action_data: { from: 'search' }

A similar event will occur when the user exits the search context by clearing the search input but with action_data: { from: 'blank-search-input' }

  • Log an event when a search is run, storing how many characters are in the search string, and the number of results -- not the actual search string.

This will be action: 'search' with action_data: { queryLength: 5, resultCount: 10 } Result count is often going to be 10 because that's the current limit we are using.

  • Log an event when a user clicks on a search result, storing the ranking of the link they clicked, but not the specific link.

This will be action: 'link-click' with action_data: 'search-result-N' Where N is the rank of the result, starting at 1

@SBisson -- thanks. This looks good to me. I think it would be good to start moving this toward testing in Test Wiki.

@kostajh and @Catrope -- can we move this toward being tested as we await approval? We want to test in Test Wiki so we can look at the events in Hadoop.

can we move this toward being tested as we await approval? We want to test in Test Wiki so we can look at the events in Hadoop.

I'll review the patch tomorrow if Roan doesn't get to it today, but we won't have it in test wiki until Monday.

Change 488935 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Help panel: search instrumentation

https://gerrit.wikimedia.org/r/488935

Change 489757 had a related patch set uploaded (by Kosta Harlan; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@wmf/1.33.0-wmf.16] Help panel: search instrumentation

https://gerrit.wikimedia.org/r/489757

Change 489757 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.33.0-wmf.16] Help panel: search instrumentation

https://gerrit.wikimedia.org/r/489757

Mentioned in SAL (#wikimedia-operations) [2019-02-11T19:16:54Z] <catrope@deploy1001> Synchronized php-1.33.0-wmf.16/extensions/GrowthExperiments/: Help panel search instrumentation (T211166) (duration: 00m 47s)

Etonkovidova closed this task as Resolved.Feb 14 2019, 6:23 PM
Etonkovidova added a subscriber: Etonkovidova.

Checked in betalabs: search-focus and search events are recorded for both desktop and mobile.

Since the detailed analysis will be done on T206719, this task is closed as Resolved.